Organic farming is believed to improve soil fertility by enhancing soil organic matter (SOM) contents. An important co-benefit would be the sequestration of carbon from atmospheric CO2. Such a positive effect has been suggested based on data from field experiments though many studies were not designed to address the issue of carbon sequestration. The aim of our study was to examine published data in order to identify possible flaws such as missing a proper baseline, carbon mass measurements, or lack of a clear distinction between conventional and organic farming practices, thereby attributing effects of specific practices to organic farming, which are not uniquely organic. A total of 68 data sets were analyzed from 32 peer-reviewed publications aiming to compare conventional with organic farming. The analysis revealed that after conversion, soil C content (SOC) in organic systems increased annually by 2.2% on average, whereas in conventional systems SOC did not change significantly. The majority of publications reported SOC concentrations rather than amounts thus neglecting possible changes in soil bulk density. 34 out of 68 data sets missed a true control with well-defined starting conditions. In 37 out of 50 cases, the amount of organic fertilizer in the organic system exceeded that applied in the compared conventional system, and in half of the cases crop rotations differed between systems. In the few studies where crop rotation and organic fertilization were comparable in both systems no consistent difference in SOC was found. From this data analysis, we conclude that the claim for beneficial effects of organic farming on SOC is premature and that reported advantages of organic farming for SOC are largely determined by higher and often disproportionate application of organic fertilizer compared to conventional farming.