The Ark Lost Island map has several gem locations. You can find the gems in three caves in the Lost Island. The first location is the best gem location in my opinion. It is in a small under the sea cave with oxygen are no nasty creatures to fight. There aren't many gems there, but you will find red, green, and blue gems. There are not a lot of items to craft on the Lost Island that require gems, so a few gems will be all you need. To get to that underwater cave you will need a scuba tank, or some swimming dino because you will run out of oxygen before you get there. A frog would be ideal because I think you could ride it through the narrow opening of the cave, and it will get you there fast enough, so you won't need a scuba tank. So far I have not seen any dangerous ocean creatures near the cave, but you never know.
Gem Location Coordinates
22.8 13.3 Red, Green, and Blue Gems - underwater cave
28.9 72.3 Blue Gems
28.4 54.0 Green, Blue, and Red Gems - aberration cave
The other two locations have all the normal and dangerous cave creatures. The last has raptors and ravagers, that can knock you off a mount. It is very dangerous but has more gems, black pearls, loot drops, and an artifact.