SOLVED: I ended up just making a tilemap that was more filled out for bitmask use. I made it just one tree tile that was copied and then had each different bitmask possible for each copy. I guess problems like these are just made when there isn't a proper bitmask.
Thanks for the response!
So I have tried variations of this before, and the problem with it is that it basically comes down to the fact that not every forest tile on the screen is technically an actual registered forest tile. I have checked this by just getting the cell of every forest (getusedcells()) and then comparing that to what I see generated on the map and it only will grab some of the forests in a column. I probably should have mentioned this in my initial post, sorry about that.
I'm not sure how this works exactly, but I think what is happening is that it will set like a "root" of the autotile and then it will fill in the rest of the autotile automatically without actually setting it as a forest cell. I don't know what it bases that off of at all; I might just have something inherently wrong with my code (maybe something with how it is bitmasked), but I can't seem to find anything blaringly wrong about it. I'll attach an example to show what I mean and hopefully someone can make sense out of it.
Here is a picture showing another map example. I marked which parts are recognized as forest tiles in the code (for just the first batch of trees at the top). Columns are the x axis and rows are the y axis.
Here is a screenshot of the code printing the used cells as well as the output for this example. It scans through the grid from left to right and then top to bottom.
So as you can see it just clearly ignores more than half of that part of the forest for whatever reason. I guess this is an entirely different problem, so that's why I figured I would just ask here if there is a way to check within an entire bitmask region, instead of this problem, with a function like withinbitmaskregion() or something. (sounds too good to be a thing though lol)
If anything else needs to be clarified just let me know because I know I could have probably explained this problem a bit better. Thanks again for the response!