Yes, it tends to be more expensive but organic growing practices are much better for the environment, and for health as well. Of course, the environment and health are connected. Folks, I just learned that workers at non-organic ("conventional") farms have died due to constant exposure to chemicals! Have a look at this discussion on Facebook for more information, websites to look at, etc.
Where to buy organic food? A lot of regular grocery stores, like my QFC at University Village, now carry some. You can also check out Whole Foods if there's one near you. Don't forget to look around you for small, local organic stores. Google it if you can't find any organic food nearby just from browsing at your regular grocery store or walking (or riding) around outside. Look for specials if you can; sometimes an organic item will be marked down to the same price as a non-organic, or even lower.
What about kosher organic? A lot of packaged items that are marked organic also have kosher certification. Of course, fresh fruits and veggies are kosher in any case, though I know some people are very careful about lettuce and such. You might have trouble finding meat that's both kosher and organic. If there's none available where you live, you can order online. (They seem to be out of stock or almost out of stock for a number of items. People have probably placed their orders already for Passover. Also it's expensive, even before shipping costs are added on. But you don't have to eat meat every day!)