the_capri_data_base
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
the_capri_data_base [2020/04/25 14:11] – matsz | the_capri_data_base [2022/02/17 13:40] – [Annex: Testing procedure and checking intermediate steps in COCO (biofuels)] peterw | ||
---|---|---|---|
Line 95: | Line 95: | ||
** COCO1 module: **\\ | ** COCO1 module: **\\ | ||
- | Prepare national database for all EU27 Member States the Western Balkan Countries, Turkey and Norway.\\ \\ It is basically divided into three main parts: | + | Prepare national database for all EU27 Member States the Western Balkan Countries, Turkey and Norway. |
+ | |||
+ | It is basically divided into three main parts: | ||
* A data import “part” that is not a single “module” but rather a collection activity to prepare a large set of very heterogeneous input files | * A data import “part” that is not a single “module” but rather a collection activity to prepare a large set of very heterogeneous input files | ||
* Including and combining these partly overlapping input data according to some hierarchical overlay criteria, and | * Including and combining these partly overlapping input data according to some hierarchical overlay criteria, and | ||
Line 109: | Line 111: | ||
** Figure 2: Overview on key elements in the consolidation of European data at the Member state level (in coco1.gms) ** | ** Figure 2: Overview on key elements in the consolidation of European data at the Member state level (in coco1.gms) ** | ||
- | {{:fig02.png? | + | {{:figure_02.png? |
Source: Own illustration | Source: Own illustration | ||
Line 121: | Line 123: | ||
** Table 3: Data items and their main sources ** | ** Table 3: Data items and their main sources ** | ||
^ Data items ^ Source ^ | ^ Data items ^ Source ^ | ||
- | | Activity levels | Eurostat: Crop production statistics, Land use statistics, herd size statistics, slaughtering statistics, statistics on import and export of live animals\\ \\ For Western Balkan Countries and Turkey: Eurostat supplemented with national statistical yearbooks, data from national ministries, FAOstat production statistics and others | | + | | Activity levels | Eurostat: Crop production statistics, Land use statistics, herd size statistics, slaughtering statistics, statistics on import and export of live animals For Western Balkan Countries and Turkey: Eurostat supplemented with national statistical yearbooks, data from national ministries, FAOstat production statistics and others | |
- | | Production, farm and market balance positions | Eurostat: Farm and market balance statistics, crop production statistics, slaughtering statistics, statistics on import and export of live animals\\ \\ For Western Balkan Countries and Turkey: Eurostat supplemented with national statistical yearbooks, data from national ministries, | + | | Production, farm and market balance positions | Eurostat: Farm and market balance statistics, crop production statistics, slaughtering statistics, statistics on import and export of live animals For Western Balkan Countries and Turkey: Eurostat supplemented with national statistical yearbooks, data from national ministries, |
- | | Sectoral revenues, costs, and producer prices | Eurostat: Economic Accounts for Agriculture (EAA) and price indices for gap filling, otherwise unit value calculation\\ \\ For Western Balkan Countries and Turkey: Supplemented with national statistical yearbooks, data from national ministries, results from AgriPolicy, FAOstat price statistics | | + | | Sectoral revenues, costs, and producer prices | Eurostat: Economic Accounts for Agriculture (EAA) and price indices for gap filling, otherwise unit value calculation For Western Balkan Countries and Turkey: Supplemented with national statistical yearbooks, data from national ministries, results from AgriPolicy, FAOstat price statistics | |
| Consumer prices | Derived from macroeconomic expenditure data (Eurostat, supplemented with UNSTATS) and food price information from various sources | | | Consumer prices | Derived from macroeconomic expenditure data (Eurostat, supplemented with UNSTATS) and food price information from various sources | | ||
| Output coefficients | Derived from production and activity levels, engineering knowledge | | | Output coefficients | Derived from production and activity levels, engineering knowledge | | ||
Line 133: | Line 135: | ||
** Table 4: Temporal coverage of national data by region ** | ** Table 4: Temporal coverage of national data by region ** | ||
^ Member State ^ Range ^ | ^ Member State ^ Range ^ | ||
- | | EU15 Member States without Germany | 1984 – 2014 | | + | | EU15 Member States without Germany | 1984 – 2018/ |
- | | Germany and (12) New Member States | 1989 – 2014 | | + | | Germany and (12) New Member States | 1989 – 2018/ |
- | | Western Balkan (WB) Countries and Turkey | 1995 – 2014 | | + | | Western Balkan (WB) Countries and Turkey | 1995 – 2018/ |
- | | Norway | 1984 – 2014 | | + | | Norway | 1984 – 2017 | |
=== Eurostat data === | === Eurostat data === | ||
Line 162: | Line 164: | ||
** Second step: data selection and code mapping ** | ** Second step: data selection and code mapping ** | ||
+ | |||
The second step is data selection and code mapping performed by the GAMS program // | The second step is data selection and code mapping performed by the GAMS program // | ||
Line 169: | Line 172: | ||
* // | * // | ||
- | Example from file // | + | Example from file // |
- | | SET AgriProdOriEurostat / \\ apro_acs_a_C1000_AR " | + | <code fortran> |
+ | SET EcoActMAP(ASS_COLS, | ||
+ | EAAP.CERE. aact_eaa01_01000_PROD_PP_MIO_EUR | ||
+ | EAAP.SWHE. | ||
+ | EAAP.DWHE. aact_eaa01_01120_PROD_PP_MIO_EUR /; | ||
- | The results of the program run are gdx-files loaded by files (e.g. coco/coco1_eurostat.gms) which are in turn loaded by coco1.gms or coco2.gms. | + | |
+ | |||
+ | SET AgriProdMAP(ASS_COLS, | ||
+ | CERE.LEVL.( apro_cpnh1_C1000_AR, | ||
+ | SWHE.LEVL.( apro_cpnh1_C1110_AR, | ||
+ | SWH1.LEVL.( apro_cpnh1_C1111_AR, | ||
+ | </ | ||
=== Western Balkan Countries and Turkey === | === Western Balkan Countries and Turkey === | ||
Line 190: | Line 203: | ||
* For all regions FAO data (mapped in the context of module “global database” to CAPRI codes and hence consistent across modules) serve as a fall back option under certain conditions, defined in the code. This fall back function of FAO data has gained in importance since Eurostat discontinued the publication of most market balances since 2014. In some cases also activity level (area) information may be taken from FAO. | * For all regions FAO data (mapped in the context of module “global database” to CAPRI codes and hence consistent across modules) serve as a fall back option under certain conditions, defined in the code. This fall back function of FAO data has gained in importance since Eurostat discontinued the publication of most market balances since 2014. In some cases also activity level (area) information may be taken from FAO. | ||
- | * Some particular data like disaggregate data on herds of chicken, ducks, turkeys and geese are compiled in a separate include file dat\coco\fao_add.gms because these data types are usually not loaded for global database. | + | * Some particular data like disaggregate data on herds of chicken, ducks, turkeys and geese are compiled in a separate include file dat/coco/fao_add.gms because these data types are usually not loaded for global database. |
=== Other additional input data === | === Other additional input data === | ||
Line 344: | Line 357: | ||
The file handling the previous actions is ‘coco1_finish_agriprod.gms’: | The file handling the previous actions is ‘coco1_finish_agriprod.gms’: | ||
- | {{:SS05.png?nolink|}} | + | {{:ss05_neu.jpg?nolink|}} |
The previous code snippet also shows for the interested reader two frequently used debugging devices: | The previous code snippet also shows for the interested reader two frequently used debugging devices: | ||
Line 350: | Line 363: | ||
- The key parameters at a certain point in the program flow (above: p_agriProd, p_agriPri, p_ecoAct) are copied to a debugging parameter “debug” (better name would be: “p_debug”). At the end of a coco1 run (or if desired also at this point) the parameter is unloaded into a file “results/ | - The key parameters at a certain point in the program flow (above: p_agriProd, p_agriPri, p_ecoAct) are copied to a debugging parameter “debug” (better name would be: “p_debug”). At the end of a coco1 run (or if desired also at this point) the parameter is unloaded into a file “results/ | ||
- The next command “$batinclude “util/ | - The next command “$batinclude “util/ | ||
- | - | + | |
Finally the biofuel sector is prepared. | Finally the biofuel sector is prepared. | ||
Line 406: | Line 419: | ||
//Fuel prices and taxes// | //Fuel prices and taxes// | ||
- | For a specification of processing-, | + | For a specification of processing-, |
===Assigning data to database array=== | ===Assigning data to database array=== | ||
Line 528: | Line 541: | ||
* To acknowledge that the Corine Classes may be mapped to several LUCAS categories we multiplied them with the “profiles”, | * To acknowledge that the Corine Classes may be mapped to several LUCAS categories we multiplied them with the “profiles”, | ||
* **LEVRegio** - Eurostat regional land use data (Eurostat Table: “agr_r_landuse”, | * **LEVRegio** - Eurostat regional land use data (Eurostat Table: “agr_r_landuse”, | ||
- | * **LEVFAO** - Land use data from the resource FAOSTAT domain FIXME ((See [[http:// | + | * **LEVFAO** - Land use data from the resource FAOSTAT domain FIXME ((See [[http:// |
* **LEVLucas** – directly using the LUCAS data is an option that has been considered but not implemented in CAPRI so this code is not used at the moment. | * **LEVLucas** – directly using the LUCAS data is an option that has been considered but not implemented in CAPRI so this code is not used at the moment. | ||
* **LEVLandCov** - Eurostat land cover data for 2009, 2012, 2015 at the MS level. Agricultural land is only distinguished into cropland CROP and grassland GRAS, but 5 nonagricultural areas are neatly aggregating up to the total country (Artificial ARTIF, shrubland (considered similar to “other wooded land” OWL), bare land & wetlands (mapped to “other sparcely vegetated or bare OSPA) and waters WATER. | * **LEVLandCov** - Eurostat land cover data for 2009, 2012, 2015 at the MS level. Agricultural land is only distinguished into cropland CROP and grassland GRAS, but 5 nonagricultural areas are neatly aggregating up to the total country (Artificial ARTIF, shrubland (considered similar to “other wooded land” OWL), bare land & wetlands (mapped to “other sparcely vegetated or bare OSPA) and waters WATER. | ||
Line 616: | Line 629: | ||
It should be mentioned that the above representation of the COCO objective function is a quite simplified one: It is evident that the above lacks safeguards against division by zero or very small values which are included in the GAMS code. Furthermore there are different types of gaps which are not reflected above to avoid clutter (Are there gaps in a series with some data or is the series empty? Is the mean based on data or estimated from \(y_{i, | It should be mentioned that the above representation of the COCO objective function is a quite simplified one: It is evident that the above lacks safeguards against division by zero or very small values which are included in the GAMS code. Furthermore there are different types of gaps which are not reflected above to avoid clutter (Are there gaps in a series with some data or is the series empty? Is the mean based on data or estimated from \(y_{i, | ||
- | Equation 4 indicates that accountancy restrictions are added. These restrictions can be balances (land, milk contents, young animals), aggregation conditions, definitions for processing coefficients and yields etc. They are quite similar to those applied for the ex ante trend projections as discussed in detail in Section [[The Regionalised Data Base (CAPREG)]] but the COCO1 accounting identities tend to acknowledge more details or have to establish the data base that is subsequently given for the ex ante trend projections, | + | Equation 4 indicates that accountancy restrictions are added. These restrictions can be balances (land, milk contents, young animals), aggregation conditions, definitions for processing coefficients and yields etc. They are quite similar to those applied for the ex ante trend projections as discussed in detail in Section [[The capri data base#The Regionalised Data Base (CAPREG)]] but the COCO1 accounting identities tend to acknowledge more details or have to establish the data base that is subsequently given for the ex ante trend projections, |
{{: | {{: | ||
Line 741: | Line 754: | ||
**Figure 3: Overview on main estimations in for the consolidation of national data in Europe (in coco1.gms)** | **Figure 3: Overview on main estimations in for the consolidation of national data in Europe (in coco1.gms)** | ||
- | {{::figure3.png?600|}} | + | {{::figure_03.png?600|}} |
Results are not always fully satisfactory (perhaps impossible given some raw data). For example the resulting prices (unit values) are far from a priori expectations for a number of series, in particular less important ones. This is because, apart from some additional security checks, unit values are by and large considered a free balancing variable calculated to preserve the identity between largely fixed EAA values and fixed production (in coco1_estimb). The priority for EAA values has been reduced somewhat in recent years but a more thorough revision would require to estimate production, market balances and EAA simultaneously rather than consecutively (first $(a)$, then $(c)$ for crops). As this is infeasible for all crops at the same time the whole estimation would need to be split up differently in the crop sector, perhaps first for the aggregates and then within those. | Results are not always fully satisfactory (perhaps impossible given some raw data). For example the resulting prices (unit values) are far from a priori expectations for a number of series, in particular less important ones. This is because, apart from some additional security checks, unit values are by and large considered a free balancing variable calculated to preserve the identity between largely fixed EAA values and fixed production (in coco1_estimb). The priority for EAA values has been reduced somewhat in recent years but a more thorough revision would require to estimate production, market balances and EAA simultaneously rather than consecutively (first $(a)$, then $(c)$ for crops). As this is infeasible for all crops at the same time the whole estimation would need to be split up differently in the crop sector, perhaps first for the aggregates and then within those. | ||
Line 814: | Line 827: | ||
& \cdot PEX_{m, | & \cdot PEX_{m, | ||
& | & | ||
- | |||
\end{split} | \end{split} | ||
\end{align} | \end{align} | ||
Line 821: | Line 833: | ||
Parameters are | Parameters are | ||
- | | \(HCOM_{m, | + | |\(HCOM_{m, |
- | | \(UVAD_{m, | + | |\(UVAD_{m, |
|\(CPS_{m, | |\(CPS_{m, | ||
|\(DCPS_{m, | |\(DCPS_{m, | ||
Line 915: | Line 927: | ||
* In some cases it is convenient to have the completed COCO1 results of all countries at hand for comparison purposes and in order to achieve a balanced picture across MS. This is the main motive for the assignments of consumer loss rates (Section 3.2.7.1). | * In some cases it is convenient to have the completed COCO1 results of all countries at hand for comparison purposes and in order to achieve a balanced picture across MS. This is the main motive for the assignments of consumer loss rates (Section 3.2.7.1). | ||
- | * Whenever averages of consolidated data (from COCO1) across several or all MS are involved, a solution in a loop requires certain sequence (such as first solving for non-candidate countries to form the averages that are input to candidate countries) or is better solved in a new module like COCO2. This applies to the expenditure allocation problem (Section [[The Complete and Consistent Data Base (COCO) for the national scale#COCO2: Data Preparation]]), | + | * Whenever averages of consolidated data (from COCO1) across several or all MS are involved, a solution in a loop requires certain sequence (such as first solving for non-candidate countries to form the averages that are input to candidate countries) or is better solved in a new module like COCO2. This applies to the expenditure allocation problem (Section [[the capri data base#COCO2: Data Preparation]]), |
===Assignment of consumer loss rates and nutrient intake per head === | ===Assignment of consumer loss rates and nutrient intake per head === | ||
- | Since a number of years diet shift scenarios have increase in importance and therefore the plausibility of per capita consumption projectios and hence their starting values, per capita consumption in the data base. A common yardstick to assess plausibility is nutrient (e.g. calorie) consumption per head where the nutrition literature offers guidance in terms of recommendable as well as “observed” consumption. For nutrition issues it is intake, so consumption after losses, which matters, such that the assignment of these loss rates becomes a critical element of the database. The starting values are due to an FAO study and stored in the \dat folder | + | Since a number of years diet shift scenarios have increase in importance and therefore the plausibility of per capita consumption projectios and hence their starting values, per capita consumption in the data base. A common yardstick to assess plausibility is nutrient (e.g. calorie) consumption per head where the nutrition literature offers guidance in terms of recommendable as well as “observed” consumption. For nutrition issues it is intake, so consumption after losses, which matters, such that the assignment of these loss rates becomes a critical element of the database. The starting values are due to an FAO study and stored in the /dat folder |
{{:: | {{:: | ||
Line 980: | Line 992: | ||
\begin{align} | \begin{align} | ||
\begin{split} | \begin{split} | ||
- | |||
& | & | ||
&= min \left[ DAYS_{up}^{HEIF}, | &= min \left[ DAYS_{up}^{HEIF}, | ||
- | |||
\end{split} | \end{split} | ||
\end{align} | \end{align} | ||
Line 1019: | Line 1029: | ||
After this last include file the completions in module COCO2 are finished and the main output file (coco2_output.gdx) is unloaded. This file is loaded in subsequent modules (main use in CAPREG, but also in CAPTRD for nowcasting and in CAPMOD for update of LULUCF coefficients). | After this last include file the completions in module COCO2 are finished and the main output file (coco2_output.gdx) is unloaded. This file is loaded in subsequent modules (main use in CAPREG, but also in CAPTRD for nowcasting and in CAPMOD for update of LULUCF coefficients). | ||
+ | |||
+ | |||
+ | |||
+ | ====Annex: Code lists for the COCO database==== | ||
+ | |||
+ | This section includes detailed code lists, which are in use in the COCO database. | ||
+ | |||
+ | **Table: Codes used for storing the original REGIO tables in the database and their description (rows)** | ||
+ | |||
+ | ^ Codes used in CAPRI REGIO tables ^ Original REGIO description ^ | ||
+ | | TOTL | Territorial area | | ||
+ | | FORE | Forest land | | ||
+ | | AGRI | Utilized agricultural area | | ||
+ | | GARD | Private gardens | | ||
+ | | GRAS | Permanent grassland | | ||
+ | | PERM | Permanent crops | | ||
+ | | VINE | Vineyards | | ||
+ | | OLIV | Olive plantations | | ||
+ | | ARAB | Arable land | | ||
+ | | GREF | Green fodder on arable land | | ||
+ | | CERE | Cereals (including rice) | | ||
+ | | WHEA | Soft and durum wheat and spelt | | ||
+ | | BARL | Barley | | ||
+ | | MAIZ | Grain maize | | ||
+ | | RICE | Rice | | ||
+ | | POTA | Potatoes | | ||
+ | | SUGA | Sugar beet | | ||
+ | | OILS | Oilseeds (total) | | ||
+ | | RAPE | Rape | | ||
+ | | SUNF | Sunflower | | ||
+ | | TOBA | Tobacco | | ||
+ | | MAIF | Fodder maize | | ||
+ | | CATT | Cattle (total) | | ||
+ | | COWT | Cows (total) | | ||
+ | | DCOW | Dairy cows | | ||
+ | | CALV | Other cows | | ||
+ | | CAT1 | Total cattle under one year | | ||
+ | | CALF | Slaughter calves | | ||
+ | | CABM | Male breeding calves (\<1 year) | | ||
+ | | CABF | Female breeding calves (\<1 year) | | ||
+ | | BUL2 | Male cattle (1-2 years) | | ||
+ | | H2SL | Slaughter heifers (1-2 years) | | ||
+ | | H2BR | Female cattle (1-2 years) | | ||
+ | | BUL3 | Male cattle (2 years and above) | | ||
+ | | H3SL | Slaughter heifers (2 years and above) | | ||
+ | | H3BR | Breeding heifers | | ||
+ | | BUFF | Total buffaloes | | ||
+ | | PIGS | Total pigs (total) | | ||
+ | | PIG1 | Piglets under 20 kg | | ||
+ | | PIG2 | Piglets under 50 kg and over 20 kg | | ||
+ | | PIG3 | Fattening pigs over 50 kg | | ||
+ | | BOAR | Breeding boars | | ||
+ | | SOW2 | Total breeding sows | | ||
+ | | SOW1 | Sows having farrowed | | ||
+ | | GILT | Gilts having farrowed for the first time | | ||
+ | | SOWM | Maiden sows | | ||
+ | | GILM | Maiden gilts | | ||
+ | | SHEP | Sheep total) | | ||
+ | | GOAT | Goats (total) | | ||
+ | | EUQI | Equidae (total) | | ||
+ | | POUL | Poultry (total) | | ||
+ | | OUTP | Final production | | ||
+ | | CROP | Total crops production | | ||
+ | | DWHE | Durum wheat | | ||
+ | | PULS | Pulses | | ||
+ | | ROOT | Roots and tubers | | ||
+ | | INDU | Industrial crops | | ||
+ | | TEXT | Textile fibre plants | | ||
+ | | HOPS | Hops | | ||
+ | | VEGE | Fresh vegetables | | ||
+ | | TOMA | Tomatoes | | ||
+ | | CAUL | Cauliflowers | | ||
+ | | FRUI | Fresh fruit | | ||
+ | | APPL | Apples | | ||
+ | | PEAR | Pears | | ||
+ | | PEAC | Peaches | | ||
+ | | CITR | Citrus fruit (total) | | ||
+ | | ORAN | Oranges | | ||
+ | | LEMN | Lemons | | ||
+ | | MAND | Mandarins | | ||
+ | | GRAP | Table grapes | | ||
+ | | WINE | Wine | | ||
+ | | TABO | Table olives | | ||
+ | | OLIO | Olive oil | | ||
+ | | NURS | Nursery plants | | ||
+ | | FLOW | Flowers and ornamental plants | | ||
+ | | OCRO | Other crops | | ||
+ | | ANIT | Total animal production | | ||
+ | | ANIM | Animal | | ||
+ | | SHGO | Sheep and goats | | ||
+ | | ANIP | Animal products | | ||
+ | | MILK | Milk | | ||
+ | | EGGS | Eggs | | ||
+ | | INPU | Intermediate consumption (total) | | ||
+ | | FEED | Animal feeding stuffs | | ||
+ | | FDGR | Animal compounds for grazing livestock | | ||
+ | | FDPI | Animal compounds for pigs | | ||
+ | | FDPO | Animal compounds for poultry | | ||
+ | | FODD | Straight feeding stuffs | | ||
+ | | FERT | Fertilizers and enrichments | | ||
+ | | ENER | Energy and lubricants | | ||
+ | | INPO | Other inputs | | ||
+ | | GVAM | Gross value added at market prices | | ||
+ | | SUBS | Subsidies | | ||
+ | | TAXS | Taxes linked to production (including VAT balance) | | ||
+ | | GVAF | Gross value added at factor costs | | ||
+ | | DEPM | Depreciation | | ||
+ | | LABO | Compensation and social security contributions of employees | | ||
+ | | RENT | Rent and other payments | | ||
+ | | INTE | Interests | | ||
+ | | GFCF | Total of gross fixed capital formation | | ||
+ | | BUIL | Buildings and other structures | | ||
+ | | MACH | Transport equipment and machinery | | ||
+ | | GFCO | Other gross fixed capital formation | | ||
+ | |||
+ | **Table: Codes used for storing the original REGIO tables in the data base and their description (columns)** | ||
+ | |||
+ | ^ Codes used in CAPRI REGIO tables ^ Original REGIO description ^ | ||
+ | | LEVL | Herd size / Area / # of persons | | ||
+ | | LSUN | Live stock units | | ||
+ | | PROP | Physical production | | ||
+ | | YILD | Yield | | ||
+ | | VALE | EAA position in ECU | | ||
+ | | VALN | EAA position in NC | | ||
+ | |||
+ | **Table: Connection between CAPRI and REGIO crop areas, crop production and herd sizes** | ||
+ | |||
+ | ^ SPEL-code ^ REGIO-code ^ REGIO-code ^ REGIO-code ^ REGIO-code ^ Description of SPEL activity ^ | ||
+ | | SWHE | WHEA | CERE | ARAB | | Soft wheat | | ||
+ | | DWHE | WHEA | CERE | ARAB | | Durum wheat | | ||
+ | | RYE | | CERE | ARAB | | Rye | | ||
+ | | BARL | BARL | CERE | ARAB | | Barley | | ||
+ | | OATS | | CERE | ARAB | | Oats | | ||
+ | | MAIZ | MAIZ | CERE | ARAB | | Maize | | ||
+ | | OCER | | CERE | ARAB | | Other cereals (excl. rice) | | ||
+ | | PARI | RICE | CERE | ARAB | | Paddy rice | | ||
+ | | PULS | | | ARAB | | Pulses | | ||
+ | | POTA | POTA | | ARAB | | Potatoes | | ||
+ | | SUGB | SUGA | | ARAB | | Sugar beet | | ||
+ | | RAPE | RAPE | OILS | ARAB | | Rape and turnip rape | | ||
+ | | SUNF | SUNF | OILS | ARAB | | Sunflower seed | | ||
+ | | SOYA | | OILS | ARAB | | Soya beans | | ||
+ | | OLIV | | OLIV | PERM | | Olives for oil | | ||
+ | | OOIL | | OILS | ARAB | | Other oil seeds and oleaginous fruits | | ||
+ | | FLAX | | | ARAB | | Flax and hemp \*\*\* (faser) \*\*\* | | ||
+ | | TOBA | TOBA | | ARAB | | Tobacco, unmanufactured, | ||
+ | | OIND | | | ARAB | | Other industrial crops | | ||
+ | | CAUL | | | ARAB | | Cauliflowers | | ||
+ | | TOMA | | | ARAB | | Tomatoes | | ||
+ | | OVEG | | | ARAB | | Other vegetables | | ||
+ | | APPL | | | PERM | | Apples, pears and peaches | | ||
+ | | OFRU | | | PERM | | Other fresh fruits | | ||
+ | | CITR | | | PERM | | Citrus fruits | | ||
+ | | TAGR | | VINE | PERM | | Table grapes | | ||
+ | | TABO | | OLIV | PERM | | Table olives | | ||
+ | | TWIN | | VINE | PERM | | Table wine | | ||
+ | | OWIN | | VINE | PERM | | Other wine | | ||
+ | | NURS | | | PERM | | Nursery plants | | ||
+ | | FLOW | | | ARAB | | Flowers, | ||
+ | | OCRO | | | ARAB | | Other final crop products | | ||
+ | | MILK | DCOW | | | | Dairy cows | | ||
+ | | BEEF | BUL2 | BUL3 | | | Bulls fattening | | ||
+ | | CALF | CALF | | | | Calves fattening (old VEAL) | | ||
+ | | PORK | PIG3 | PIG2 | PIG1 | | Pig fattening | | ||
+ | | MUTM | GOAT | SHEP | | | Ewes and goats | | ||
+ | | MUTT | GOAT | SHEP | | | Sheep and goat fattening | | ||
+ | | EGGS | POUL | | | | Laying hens | | ||
+ | | POUL | POUL | | | | Poultry fattening | | ||
+ | | OANI | | | | | Other animals | | ||
+ | | OROO | | | ARAB | | Other root crops | | ||
+ | | GRAS | GRAS | | | | Green fodder | | ||
+ | | SILA | GREF | | ARAB | | Silage | | ||
+ | | CALV | CALV | | | | Suckler cows | | ||
+ | | RCAL | CABM | CABF | | | Calves, raising | | ||
+ | | HEIF | H2SL | H2BR | H3SL | H3BR | Heifers | | ||
+ | | PIGL | SOW2 | | | | Pig breeding | | ||
+ | | FALL | | | FALL | | Fallow land | | ||
+ | |||
+ | **Tables: Codes of the input allocation estimation** | ||
+ | |||
+ | ^FADN inputs (FI) ^Label | ||
+ | | TOIN | total inputs | | ||
+ | | COSA | animal specific inputs | | ||
+ | | FEDG | self grown feedings | | ||
+ | | ANIO | other animal inputs | | ||
+ | | FEDP | purchased feedings | | ||
+ | | COSC | crop specific inputs | | ||
+ | | SEED | seeds | | ||
+ | | PLAP | plant protection | | ||
+ | | FERT | fertilisers | | ||
+ | | TOIX | other inputs (overheads) | | ||
+ | |||
+ | |||
+ | ^CAPRI inputs (CI) used in the reconciliation ^label ^ | ||
+ | | TOIN | total inputs | | ||
+ | | FEED | feedings | | ||
+ | | IPHA | other animal inputs | | ||
+ | | COSC | crop specific inputs | | ||
+ | | SEED | seeds | | ||
+ | | PLAP | plant protection | | ||
+ | | FERT | fertilisers | | ||
+ | | REPA | repairs | | ||
+ | | ENER | energy | | ||
+ | | SERI | agricultural services input | | ||
+ | | INPO | other inputs | | ||
+ | |||
+ | 1. The set of //Other// activities that had been omitted from the econometric estimation: | ||
+ | |||
+ | * OTHER={OCER, | ||
+ | |||
+ | 2. The set of activity groups, and their elements, used in the replacement or missing/ | ||
+ | |||
+ | * GROUPS = {YOUNG, VEGE, SETT, PULS, PIG, OILS, MILK, MEAT, INDS, HORSE, GOAT, FRU, FOD, FLOWER, DENNY, COW, CHICK1, CHICK2, CHICK3, CERE, ARAB} | ||
+ | * YOUNG={YBUL, | ||
+ | * VEGE={TOMA}, | ||
+ | * SETT={SETA, NONF, FALL, GRAS}, | ||
+ | * PULS=PULS | ||
+ | * PIG={PIGF, SOWS}, | ||
+ | * OILS={RAPE, SOYA, SUNF, PARI, OLIV}, | ||
+ | * INDS={TOBA, TEXT, TABO}, | ||
+ | * GOAT={SHGM, SHGF}, | ||
+ | * FRU={APPL, CITR, TAGR, TWIN}, | ||
+ | * FOD={ROOF, MAIF}, | ||
+ | * FLOWER={FLOW, | ||
+ | * DENNY={PORK, | ||
+ | * COW={DCOW, SCOW, HEIF, HEIR, CAMF, CAFF, BULF, CAMR, CAFR}, | ||
+ | * CHICK1={HENS, | ||
+ | * CERE={SWHE, DWHE, BARL, OATS, RYEM, MAIZ}, | ||
+ | * ARAB={POTA, SUGB} | ||
+ | |||
+ | 3. The sets of Northern European, Southern European countries: | ||
+ | |||
+ | * NEUR={NL000, | ||
+ | * SEUR={El000, | ||
+ | |||
+ | |||
+ | ** Table: Codes of land use classes (Set LandUse)** | ||
+ | |||
+ | ^Code ^Label | ||
+ | | OART | artificial | | ||
+ | | ARAO | (other) arable crops - all arable crops excluding rice and fallow (see also definition of ARAC below) | | ||
+ | | PARI | paddy rice (already defined) | | ||
+ | | GRAT | temporary grassland (alternative code used for CORINE data, definition identical to TGRA | | ||
+ | | FRCT | fruit and citrus | | ||
+ | | OLIVGR | Olive Groves | | ||
+ | | VINY | vineyard (already defined) | | ||
+ | | NUPC | nursery and permanent crops (Note: the aggregate PERM also includes flowers and other vegetables | | ||
+ | | BLWO | board leaved wood | | ||
+ | | COWO | coniferous wood | | ||
+ | | MIWO | mixed wood | | ||
+ | | POEU | plantations (wood) and eucalyptus | | ||
+ | | SHRUNTC | shrub land - no tree cover | | ||
+ | | SHRUTC | shrub land - tree cover | | ||
+ | | GRANTC | Grassland - no tree cover | | ||
+ | | GRATC | Grassland - tree cover | | ||
+ | | FALL | fallow land (already defined) | | ||
+ | | OSPA | other sparsely vegetated or bare | | ||
+ | | INLW | inland waters | | ||
+ | | MARW | marine waters | | ||
+ | | KITC | kitchen garden | | ||
+ | |||
+ | |||
+ | ** Table: Codes of land use aggregates (Set LandUseAgg)** | ||
+ | |||
+ | ^Code ^Label | ||
+ | | OLND | other land - shrub, sparsely vegetated or bare | | ||
+ | | ARAC | arable crops | | ||
+ | | FRUN | fruits, nursery and (other) permanent crops | | ||
+ | | WATER | inland or marine waters | | ||
+ | | ARTIF | artificial - buildings or roads | | ||
+ | | OWL | other wooded land - shrub or grassland with tree cover (definition to be discussed) | | ||
+ | | TWL | total wooded land - forest + other wooded land | | ||
+ | | SHRU | shrub land | | ||
+ | | FORE | forest (already defined) | | ||
+ | | GRAS | grassland (already defined) | | ||
+ | | ARAB | arable (already defined) | | ||
+ | | PERM | permanent crops (already defined) | | ||
+ | | UAAR | utilizable agricultural area (already defined) | | ||
+ | | ARTO | total area - total land and inland waters | | ||
+ | | ARTM | total area including marine waters | | ||
+ | | CROP | crop area - arable and permanent | | ||
+ | |||
+ | **Table: Codes of mutually exclusive subset adding up to total area - ARTO (Set LandUseARTO)** | ||
+ | |||
+ | ^Code ^Label | ||
+ | | OLND | other land - shrub, sparsely vegetated or bare | | ||
+ | | ARTIF | artificial - buildings or roads | | ||
+ | | FORE | forest | | ||
+ | | UAAR | utilizable agricultural area | | ||
+ | | INLW | Inland waters | | ||
+ | ====Annex: Detailed description of Eurostat data processing in COCO (coco1_eurostat.gms)==== | ||
+ | |||
+ | The program starts by importing pre-processed data from Eurostat. The pre-processing includes simple data selection routines and also manual checks. The Eurostat domains are processed one by one, and the corrections are done for each Member State ((Eurostat offers data for Belgium and Luxembourg separately, whereas the database combines both countries to the model region " | ||
+ | |||
+ | Below we discuss the specific data-processing tasks related to Eurostat table groups. | ||
+ | The first Eurostat Table Group is “p_AgriProd” covering market balances and activity levels. | ||
+ | |||
+ | // | ||
+ | |||
+ | |||
+ | * The following | ||
+ | * DWH1, RAP1, POT1, POT2, ROO1 and ROO2 are not longer supported | ||
+ | * data for slaughter heads and slaughter tons for calves are only available for recent years | ||
+ | * deliveries to dairy of RMLK missing for earlier years in selection starting with February 2018 | ||
+ | For an Interim solution, data for the missing data points are collected from an earlier Eurostat selection (March 2010). | ||
+ | * UNFCCC data is included, here sheep and goats population, to prolong data of some countries where Eurostat data collection stopped 2008/2009. | ||
+ | * Recent dairy sector data from Eurostat via DG supplements the ordinary dairy data downloaded from the website of Eurostat. | ||
+ | * Sugar trade data from the market balances of Eurostat is extended with Comext (Eurostat) data. | ||
+ | * For the milk products WMIO, SMIP, FRMI and COCM some market balance positionpositions are corrected: “industrial use” is added to “feed on market and “processing” is added to “human consumption. | ||
+ | * COCO code " | ||
+ | * All activities for the aggregate ILAM are added up from SHEP and GOAT. | ||
+ | * The units for wine balance sheets are converted from 1000hl to 10000hl=1000000l | ||
+ | * A rice milled equivalent balance without paddy rice (separate product) is constructed. | ||
+ | * Survey data on buffaloes are used to increase the bovine stock data to cover the whole cattle herd. | ||
+ | |||
+ | // | ||
+ | |||
+ | Due to years of database updates, a number of corrections on input data are carried out. For special cases in some MS, data are read in from additional data sources: | ||
+ | |||
+ | * Belgium-Luxemburg: | ||
+ | * France: market balances for cereal products (Agreste, Direction générale des douanes et droits indirects (DGDDI)) | ||
+ | * Denmark: market balances for some cereal products (StatBank Denmark) | ||
+ | * Finland: market balances for some cereal products (Natural Resources Institute Finland, Balance sheet for food commodities) | ||
+ | * Germany: activity levels for textile crops (BMELF) | ||
+ | * Ireland: trade for citrus fruits and some milk products (Eurostat: EU trade since 1988 by HS2-HS4 [DS-016894]) and activity levels for grass land (StatBank Ireland) | ||
+ | * Austria: production of cow milk, fruit products and potatoes (Statistisches Amt Österreich) | ||
+ | * Czechia: trade of life animals (Eurostat: EU trade since 1988 by CN8 [DS-016890]) | ||
+ | * Lithuania: human consumption cereal products (calculated from data from statistical yearbook 2018) | ||
+ | * Slovenia: slaughtering (SiStat Slovenia) | ||
+ | * Romania: data for the meat and in the milk sectors (Romanian experts) | ||
+ | * Trade data for sugar are collected from Eurostat COMEXT data. | ||
+ | |||
+ | |||
+ | The remaining domains/ | ||
+ | |||
+ | * The second Eurostat Table Group is “p_ExchRate” covering exchange rates | ||
+ | * The third Eurostat Table Group is “p_EcoAct” covering the economic accounts for agriculture. | ||
+ | * The fourth Eurostat Table Group is “p_AgriPri” covering agricultural producer prices. | ||
+ | |||
+ | |||
+ | |||
+ | ====Annex: Testing procedure and checking intermediate steps in COCO (biofuels)==== | ||
+ | |||
+ | The COCO module produces various reporting files on the intermediate data processing steps. These files can be used to trace back potential errors in the COCO database to their origin. These debugging files also contain meta-information on the input data and settings used for producing the COCO database. | ||
+ | |||
+ | The following example is a walk-through on the typical data processing steps, covring biofuels data preparation in France. | ||
+ | |||
+ | The reporting file ' | ||
+ | |||
+ | {{coco_biof_1.png? | ||
+ | |||
+ | The set // | ||
+ | |||
+ | {{coco_biof_2.png? | ||
+ | |||
+ | The complete list of the biofuel items in // | ||
+ | |||
+ | | bioECere | Ethanol processed from cereals | | ||
+ | | bioESuga | Ethanol processed from sugar beets | | ||
+ | | bioETwin | Ethanol processed from wine | | ||
+ | | bioEFrui | Ethanol processed from fruits | | ||
+ | | bioEOcro | Ethanol processed from other agricultural crops | | ||
+ | | bioEExog | Ethanol processed from crops not explicit in biofuel modelling (fruits, potatoes, other crops) | | ||
+ | | bioARES | Biofuels processed from crops residues | | ||
+ | | bioORES | Biofuels processed from forest residues and waste material (municipal waste, waste oil, other waste) | | ||
+ | | SECG | Biofuel quantities from second generation | | ||
+ | | MAPRagr | Ethanol production from agricultural sources | | ||
+ | | EloBio | Biofuel production and demand data from DG Energy project EloBio | | ||
+ | | DG_Agri | Ethanol data from DGAgri website and supplementary files | | ||
+ | | ProdCom | Eurostat: PRODCOM ANNUAL SOLD (NACE Rev. 2.) [DS-066341] | | ||
+ | | EIA | Independent Statistics & Analysis, US Energy Information Administration | | ||
+ | | comext | Eurostat: Comext | | ||
+ | | Energy_bal | Eurostat: Supply, transformation, | ||
+ | | Energy_dem | Eurostat: Supply, transformation, | ||
+ | | final | results of the calculations | | ||
+ | | ODOM | other domestic use (activity from biostock calculations | | ||
+ | | INDt | Sum of model results for BIOF and INDM | | ||
+ | | BIOi, INDi, DOMi | intermediate activities to save data from model initialisation for later documentation. | | ||
+ | |||
+ | |||
+ | Biofuels production (levels) are calculated for biodiesel (BIOD) and bioethanol (BIOE). Input data and final initialization values before the consistency models are run are documented on the parameter // | ||
+ | |||
+ | |||
+ | |||
+ | **Example 1: Bioethanol** | ||
+ | |||
+ | The screenshot demonstrates the input data and final initialization values collected on parameter // | ||
+ | |||
+ | {{coco_biof_3.png? | ||
+ | |||
+ | The results of the model m_bioFitE (BIOE) are documented on the parameter p_biofDatatMS. | ||
+ | |||
+ | {{coco_biof_4.png? | ||
+ | |||
+ | We take soft wheat (SWHE) as an example for biofuel feedstock, and walk through the initialization and consistency model results. From data input (Eurostat and FAO) we received in 2002 an industrial use of 894 1000t, saved on INDi. For production of bio-ethanol 631 1000t were initialized, | ||
+ | |||
+ | |||
+ | |||
+ | **Example 2: Biodiesel** | ||
+ | |||
+ | |||
+ | The first dimension of the reporting parameter // | ||
+ | The second dimension of the parameter shows the activity. | ||
+ | |||
+ | {{coco_biof_5.png? | ||
+ | |||
+ | For Bio-diesels, | ||
+ | |||
+ | ^Data source code ^Data source description ^ | ||
+ | |Primes |PRIMES MODEL, EC3MLAB of ICCS, National University of Athens| | ||
+ | |||
+ | {{coco_biof_6.png? | ||
+ | |||
+ | The parameter // | ||
+ | |||
+ | |||
+ | |||
+ | ====Annex: Testing procedure and checking intermediate steps in COCO (dairy)==== | ||
+ | |||
+ | The following three examples show how to use the intermediate reporting files to trace the data preparation steps. Screenshots demonstrate the arrangement of the reporting parameters by using the CAPRI Graphical User Interface. COCO automatically produces the reporting files in the folder // results/ | ||
+ | |||
+ | |||
+ | **Example 1: Production of cow (COMI) and sheep (SGMI) milk** | ||
+ | |||
+ | In order to document the procedure of data consolidation and rebooking, we look at the reporting file for France “chk_estima_FR000.gdx”. | ||
+ | |||
+ | {{: | ||
+ | |||
+ | The codes in the rows show the activity code, the product code and its status. For activity codes see Annex 1: Code list. | ||
+ | |||
+ | Status codes: | ||
+ | |||
+ | * INI: | ||
+ | * COCO1: | ||
+ | |||
+ | The initialization of the production of COMI and SGMI is done in the module // | ||
+ | |||
+ | * COMI: Milk from cows (CMLK) and buffaloes (BMLK) is added up. | ||
+ | * SGMI: Milk from ewes (EMLK) and goats (GMLK) is added up. | ||
+ | * If data on cow or sheep and goat milk is not available separately, but total milk production (RMLK) is available, then production of COMI is set equal to total milk production. | ||
+ | * Only COMI and SGMI are included in the estimation in // | ||
+ | * The production of RMLK and its components CMLK, BMLK, EMLK and GMLK are only copied from raw data tables into this check parameter for documentation purposes. | ||
+ | |||
+ | |||
+ | ** Example 2: data consolidation for cow milk** | ||
+ | |||
+ | The procedure of data consolidation and rebooking of all activities for the CAPRI product “COMI” (cow milk) is shown in the following screenshot (only part of the reporting parameter p_estimAnimMS is shown, but the full scope of the table is visible in the GUI). | ||
+ | |||
+ | {{: | ||
+ | |||
+ | |||
+ | The codes in the rows show the activity code and its status. For activity codes see Annex 1: Code list. | ||
+ | Additional codes for status include the following. | ||
+ | |||
+ | ^Status code | ||
+ | |StdeData |Final (small) Stde (standard deviation) attached to priors from raw data | | ||
+ | |StdeScale |Final (large) Stde attached to priors from trends but not from raw data | | ||
+ | |Upplim |Soft upper limits triggering extra penalties if violated | ||
+ | |Lowlim |Soft lower limits triggering extra penalties if violated | ||
+ | |Supps |Prior value = support: comes from raw data or trends plus HP filter | ||
+ | |Err2rev |Original error term from preest: to steer speed of bound opening | ||
+ | |||
+ | |||
+ | Under activity dairy cows (DCOW) the following items are reported: yield, total production (GROF), feed use (FEDM) and losses on market (LOSM). Eurostat’s //National Accounts of Agriculture (EAA)// only supply data for the aggregate milk (MILK). The equation // | ||
+ | |||
+ | <code fortran> | ||
+ | e_EAAMLK(" | ||
+ | $ (p_NobsP(" | ||
+ | (p_NobsP(" | ||
+ | * | ||
+ | v_EstimY(" | ||
+ | | ||
+ | + v_EstimY(" | ||
+ | |||
+ | </ | ||
+ | |||
+ | Finally. the producer prices (UVAP) are calculated directly from the monetary EEA values and production. | ||
+ | The following picture shows the data processing steps (states) for the EAA values for milk. | ||
+ | |||
+ | {{: | ||
+ | |||
+ | From the example for COMI above you can also understand the influence of the standard deviation from raw data (e.g. FEDM.StdeData), | ||
+ | |||
+ | The first initialisation of // | ||
+ | |||
+ | |||
+ | |||
+ | **Example 3: data consolidation for cow dairy cow activity (DCOW)** | ||
+ | |||
+ | The procedure of data consolidation and booking intermediate data processing results for the dairy cow activity (DCOW) is demonstrated in the following screenshot. | ||
+ | |||
+ | {{: | ||
+ | |||
+ | The rows of the table show the product item code for the production activity DCOW, and the data processing steps (status). The first two lines show the coco1 results for slaughtering. The items starting with Y and I stand for the output and input of calves. The initialization, | ||
+ | |||
===== The Regionalised Data Base (CAPREG) ===== | ===== The Regionalised Data Base (CAPREG) ===== | ||
Line 1051: | Line 1556: | ||
In the last major update of 2015 the original data had been first stored in the TSV format designed by EUROSTAT: | In the last major update of 2015 the original data had been first stored in the TSV format designed by EUROSTAT: | ||
* Unordered List ItemIn a first step, these files had been converted by an excel macro into csv format and an overall set with all items including their long text has been created to prepare further processing. | * Unordered List ItemIn a first step, these files had been converted by an excel macro into csv format and an overall set with all items including their long text has been created to prepare further processing. | ||
- | * In a second step these alredy GAMS readable files are stored in GDX format in folder “dat\capreg” and under version control. Meta data are added in the process as well. | + | * In a second step these alredy GAMS readable files are stored in GDX format in folder “dat/capreg” and under version control. Meta data are added in the process as well. |
Line 1145: | Line 1650: | ||
The regional database modules also cover some aspects which are discussed in other parts of this documentation. | The regional database modules also cover some aspects which are discussed in other parts of this documentation. | ||
- | * For policy data at the regional level (mostly premium related data) see Section [[Policy data]]. These policy related assignments require a good part of the CAPREG module | + | * For policy data at the regional level (mostly premium related data) see Section [[the capri data base#Policy data]]. These policy related assignments require a good part of the CAPREG module |
- | * For the fertiliser and feed allocations and environmental indicators, also important elements of the regional database, see the next Section [[Input Allocation]] | + | * For the fertiliser and feed allocations and environmental indicators, also important elements of the regional database, see the next Section [[the capri data base#Input Allocation]] |
* Towards the end of the regional data base consolidation supply side PMP parameters are calibrated as a final test of consistency and sometimes to serve as starting values for the subsequent baseline calibration (in // | * Towards the end of the regional data base consolidation supply side PMP parameters are calibrated as a final test of consistency and sometimes to serve as starting values for the subsequent baseline calibration (in // | ||
Line 1262: | Line 1767: | ||
|//Sum of slaughtered cows and stock change// | |//Sum of slaughtered cows and stock change// | ||
|GROFYCOW| Numer of heifers raised to young cows| 235, | |GROFYCOW| Numer of heifers raised to young cows| 235, | ||
- | |HEIRLEVL| Activity level of the heifers raising process |235, | + | |HEIRLEVL| Activity level of the heifers raising process |235, |
- | \\ Source: CAPRI Modelling System | + | |
Line 1273: | Line 1777: | ||
|Dairy cows (DCOW) |DCOL: | |Dairy cows (DCOW) |DCOL: | ||
|Bull fattening (BULF) |BULL: | |Bull fattening (BULF) |BULL: | ||
- | |Heifers fattening (HEIF)| HEIL: | + | |Heifers fattening (HEIF)| HEIL: |
- | \\ Source: CAPRI Modelling System | + | |
====Input allocation for feed==== | ====Input allocation for feed==== | ||
Line 1702: | Line 2205: | ||
| **Ex-post** | | **Ex-post** | ||
- | |**Given:**\\ -Herd sizes\\ | + | |**Given: |
+ | |-Herd sizes|-Herd sizes| | ||
+ | | => Manure output|=> manure output| | ||
+ | |-Crop areas and yields|-Crop areas and yields| | ||
+ | |=> Export with harvest|=> Export with harvest| | ||
+ | |-National anorganic application|-National and Regional anorganic application| | ||
+ | |**Estimated: | ||
+ | |-Regional anorganic application|-Factor for Fertilization beyond export (trended)| | ||
+ | |-Factor for Fertilization beyond N export|-Manure availability (trended)| | ||
+ | |-Manure availability| | ||
A good overview on how the Nitrogen balances are constructed and can be used for analysis can be found in: Leip A., Britz W., de Vries W. and Weiss F. (2011): Farm, land, and soil nitrogen budgets for agriculture in Europe calculated with CAPRI, Environmental Pollution 159(11), 3243-3253 and Leip, A., Weiss, F. and Britz, W. (2011): Agri-Environmental Nitrogen Indicators for EU27, in: Flichman G. (ed.), Bio-Economic Models applied to Agricultural Systems, p. 109-124, Springer, Netherlands. | A good overview on how the Nitrogen balances are constructed and can be used for analysis can be found in: Leip A., Britz W., de Vries W. and Weiss F. (2011): Farm, land, and soil nitrogen budgets for agriculture in Europe calculated with CAPRI, Environmental Pollution 159(11), 3243-3253 and Leip, A., Weiss, F. and Britz, W. (2011): Agri-Environmental Nitrogen Indicators for EU27, in: Flichman G. (ed.), Bio-Economic Models applied to Agricultural Systems, p. 109-124, Springer, Netherlands. | ||
Line 1872: | Line 2384: | ||
|::: | |::: | ||
|::: |Rice production|CH4Ric| | |::: |Rice production|CH4Ric| | ||
- | |::: |Land use change emissions from\\ biomass burning|CH4bur| | + | |::: |Land use change emissions from biomass burning|CH4bur| |
|**Nitrous Oxide**|Manure management|N2OMan| | |**Nitrous Oxide**|Manure management|N2OMan| | ||
|::: | |::: | ||
Line 1878: | Line 2390: | ||
|::: | |::: | ||
|::: |Crop residues|N2OCro| | |::: |Crop residues|N2OCro| | ||
- | |::: | + | |::: |
- | |::: | + | |::: |
|::: | |::: | ||
- | |::: |Land use change emissions from the \\ burning of biomass|N2Obur| | + | |::: |Land use change emissions from the burning of biomass|N2Obur| |
|**Carbon dioxide**|Cultivation of histosols|CO2his| | |**Carbon dioxide**|Cultivation of histosols|CO2his| | ||
|::: | |::: | ||
|::: | |::: | ||
- | |::: |Land use change emissions from above \\ and below ground biomass|CO2bio| | + | |::: |Land use change emissions from above and below ground biomass|CO2bio| |
- | |::: |Land use change emissions from soil \\ carbon changes|CO2soi| \\ Source: CAPRI Modelling System | + | |::: |Land use change emissions from soil carbon changes|CO2soi| \\ Source: CAPRI Modelling System |
For a detailed analysis of these single emission sources refer to Pérez 2006: Greenhouse Gases: Inventories, | For a detailed analysis of these single emission sources refer to Pérez 2006: Greenhouse Gases: Inventories, | ||
Line 1958: | Line 2470: | ||
| Region | | Region | ||
- | |:::| | regional | + | |:::| | regional |
|Belgium (BL24)|Soft wheat| 31.49| 31.26| 31.49| 24.99| 32.73| 53.88| | |Belgium (BL24)|Soft wheat| 31.49| 31.26| 31.49| 24.99| 32.73| 53.88| | ||
|:::|Sugar beet | | |:::|Sugar beet | | ||
Line 1997: | Line 2509: | ||
The head section of the sub-module comprises (a) initialization of FAOSTAT-related and mapping sets which are used in all futher consolidation sections, (b) loading union sets from the CommodityBalances and ProductionAndRessources data files, (c) introducing the land categories relevant for the land use consolidation, | The head section of the sub-module comprises (a) initialization of FAOSTAT-related and mapping sets which are used in all futher consolidation sections, (b) loading union sets from the CommodityBalances and ProductionAndRessources data files, (c) introducing the land categories relevant for the land use consolidation, | ||
- | {{:: | + | {{:: |
The first consolidation section is on “Production and Ressources”. After loading the raw data at the beginning, the FAOSTAT units are mapped to CAPRI units via the “unit_map” set and corresponding multiplication factors as provided under (d) in the head section of the program to harmonise the units. After that the data is checked for completeness and various heuristic rules are applied to fill gaps in the data: \\ | The first consolidation section is on “Production and Ressources”. After loading the raw data at the beginning, the FAOSTAT units are mapped to CAPRI units via the “unit_map” set and corresponding multiplication factors as provided under (d) in the head section of the program to harmonise the units. After that the data is checked for completeness and various heuristic rules are applied to fill gaps in the data: \\ | ||
- | {{: | + | {{: |
After aggregating data for China and some reporting on missing data the consolidated production data is written to the /fao folder in the restart-directory for usage in the following consolidation steps. | After aggregating data for China and some reporting on missing data the consolidated production data is written to the /fao folder in the restart-directory for usage in the following consolidation steps. | ||
Line 2007: | Line 2519: | ||
The next stept consolidates “Commodity Balances” and introduces the sets for the main balance components and demand positions as well as the mapping between the original FAOSTAT item codes and the commodity balance codes. This is another example that any data consolidation combining different data sets (even when coming from the same agency like FAO) needs to consider different coding systems used in those data sets: \\ | The next stept consolidates “Commodity Balances” and introduces the sets for the main balance components and demand positions as well as the mapping between the original FAOSTAT item codes and the commodity balance codes. This is another example that any data consolidation combining different data sets (even when coming from the same agency like FAO) needs to consider different coding systems used in those data sets: \\ | ||
- | {{: | + | {{: |
In addition to the item code and unit matching and the removal of flags, negative observations are removed (except for stock changes) from the data. Gap filling is based on weighted averages and smoothed interpolation. Total demand is added up from single demand positions if missing and single demand positions are scaled to given total demand in case they do not sum up consistently. Finally, stock changes are adjusted to ensure that market balances are closed. The consolidated commodity balance data is written to the /fao-folder in the restart directory for further usage inside the fao_balance_consolidation. | In addition to the item code and unit matching and the removal of flags, negative observations are removed (except for stock changes) from the data. Gap filling is based on weighted averages and smoothed interpolation. Total demand is added up from single demand positions if missing and single demand positions are scaled to given total demand in case they do not sum up consistently. Finally, stock changes are adjusted to ensure that market balances are closed. The consolidated commodity balance data is written to the /fao-folder in the restart directory for further usage inside the fao_balance_consolidation. | ||
Line 2013: | Line 2525: | ||
The next stept combines production and ressources with the data on commodity balances in order to consolidate the land use data. The consolidation procedure for land use categories is a separate sub-routine included under this section:\\ | The next stept combines production and ressources with the data on commodity balances in order to consolidate the land use data. The consolidation procedure for land use categories is a separate sub-routine included under this section:\\ | ||
- | {{: | + | {{: |
The land use consolidation step takes care of the mapping between FAOSTAT and CAPRI land use categories, imposes gap filling routines, introduces auxiliary data from UNFCCC and UNSTATS and ensures that nested land use categories consistently sum up to their totals. | The land use consolidation step takes care of the mapping between FAOSTAT and CAPRI land use categories, imposes gap filling routines, introduces auxiliary data from UNFCCC and UNSTATS and ensures that nested land use categories consistently sum up to their totals. | ||
Line 2019: | Line 2531: | ||
The land use consistency is solved as an optimization problem ensuring (a) adding up of single crop areas to land use aggegates and (b) imposes constraints stemming from transition probabilities between different UNFCCC land use categories: \\ | The land use consistency is solved as an optimization problem ensuring (a) adding up of single crop areas to land use aggegates and (b) imposes constraints stemming from transition probabilities between different UNFCCC land use categories: \\ | ||
- | {{: | + | {{: |
Finally, crop area levels are rescaled based on the solution from the optimization problem and yields are recalculated accordingly. The consolidated land use data is written to the /fao-folder in the restart directory. | Finally, crop area levels are rescaled based on the solution from the optimization problem and yields are recalculated accordingly. The consolidated land use data is written to the /fao-folder in the restart directory. | ||
Line 2031: | Line 2543: | ||
The consolidation of trade flows is split up across product specific groups to keep the task feasible in terms of computational complexity. The task is split up among 29 groups in total:\\ | The consolidation of trade flows is split up across product specific groups to keep the task feasible in terms of computational complexity. The task is split up among 29 groups in total:\\ | ||
- | {{: | + | {{: |
The whole procedure for creating a consistent data base as a starting point for the CAPRI task “Build global database” consists of two major tasks that are called the “groupSpecific” and “nongroupSpecific” tasks. The first one is the actual consolidation part that is done for each commodity group separately but executed in parallel. The second one is necessary for exporting the results such that they may be exploited via the GUI or be used as major input for the GLOBAL module. \\ | The whole procedure for creating a consistent data base as a starting point for the CAPRI task “Build global database” consists of two major tasks that are called the “groupSpecific” and “nongroupSpecific” tasks. The first one is the actual consolidation part that is done for each commodity group separately but executed in parallel. The second one is necessary for exporting the results such that they may be exploited via the GUI or be used as major input for the GLOBAL module. \\ | ||
- | {{: | + | {{: |
The group specific task starts 29 separate consolidation processes in parallel where the actual consolidation processes are defined in the separate include file “/ | The group specific task starts 29 separate consolidation processes in parallel where the actual consolidation processes are defined in the separate include file “/ | ||
The trade consolidation part requires specific FAOSTAT trade data related sets that are loaded at the beginning of the include file. There are 18 different types of output reported in the result array.\\ | The trade consolidation part requires specific FAOSTAT trade data related sets that are loaded at the beginning of the include file. There are 18 different types of output reported in the result array.\\ | ||
- | {{:: | + | {{:: |
There are also 25 different statistics reported for the time series that are important intermediate indicators for the trade consolidation process. \\ | There are also 25 different statistics reported for the time series that are important intermediate indicators for the trade consolidation process. \\ | ||
- | {{: | + | {{: |
The trade consolidation consists of eight steps in sequence that are dependent on each other, i.e. each step produces an intermediate output file that is written to the /fao folder in the restart directory for usage in the follow-up steps. | The trade consolidation consists of eight steps in sequence that are dependent on each other, i.e. each step produces an intermediate output file that is written to the /fao folder in the restart directory for usage in the follow-up steps. | ||
Line 2054: | Line 2566: | ||
In the following step (4) STATRADE a trust indicator is computed that allows to assign a trade flow value in case of conflicting notifications between trade partners. It is based on the sum of absolute differences to partner notifications relative to total notified trade. | In the following step (4) STATRADE a trust indicator is computed that allows to assign a trade flow value in case of conflicting notifications between trade partners. It is based on the sum of absolute differences to partner notifications relative to total notified trade. | ||
- | {{: | + | {{: |
The next step (5) TRDTRADE calculates national linear trend lines for quantities, values, unit values and price indices. | The next step (5) TRDTRADE calculates national linear trend lines for quantities, values, unit values and price indices. | ||
Line 2060: | Line 2572: | ||
Step (6) INITRADE prepares the trade data for the final consolidation procedure by calculating expected means of imports, exports and unit values, and by computing the trust indicator, standard errors and expected standard errors for trade quantity and units, and unit values. The trust indicator is used for adjusting the standard errors in the estimation of trade flows between partners. Higher trust indicators result in lower standard errors and lower standard errors lead to smaller deviations from reported trade, i.e. the outcome from the estimation will deviate less from the reportings for more trustworthy partners, and vice versa. | Step (6) INITRADE prepares the trade data for the final consolidation procedure by calculating expected means of imports, exports and unit values, and by computing the trust indicator, standard errors and expected standard errors for trade quantity and units, and unit values. The trust indicator is used for adjusting the standard errors in the estimation of trade flows between partners. Higher trust indicators result in lower standard errors and lower standard errors lead to smaller deviations from reported trade, i.e. the outcome from the estimation will deviate less from the reportings for more trustworthy partners, and vice versa. | ||
- | {{: | + | {{: |
The computations are accomplished for each commodity separately. | The computations are accomplished for each commodity separately. | ||
Line 2070: | Line 2582: | ||
The second “nongroupSpecific” task in the trade consolidation part takes care of exporting the consolidated trade data to the /fao-folder in the results directory. This output is a major input for the CAPRI task “Build global database” (“fao_trade_for_global…gdx”). The trade data is complemented with data on conversion coefficients, | The second “nongroupSpecific” task in the trade consolidation part takes care of exporting the consolidated trade data to the /fao-folder in the results directory. This output is a major input for the CAPRI task “Build global database” (“fao_trade_for_global…gdx”). The trade data is complemented with data on conversion coefficients, | ||
- | {{: | + | {{: |
====Task: Build global database==== | ====Task: Build global database==== | ||
Line 2078: | Line 2590: | ||
**Figure 9: Overview on key elements in the consolidation of global data (in global_database.gms)** | **Figure 9: Overview on key elements in the consolidation of global data (in global_database.gms)** | ||
- | {{:figure_8_1_.png? | + | {{:figure_09.png? |
The program starts with including three general programs also present (possibly in task specific form) in other main programms plus the steering file (runglobal.gms) with more precise settings for the current run which may come from the GUI or from a batch file((A batch file is a steering file to execute a CAPRI task with all settings that are usually made in the GUI (say which simulation years) expressed equivalently in a certain language in a text file.)): | The program starts with including three general programs also present (possibly in task specific form) in other main programms plus the steering file (runglobal.gms) with more precise settings for the current run which may come from the GUI or from a batch file((A batch file is a steering file to execute a CAPRI task with all settings that are usually made in the GUI (say which simulation years) expressed equivalently in a certain language in a text file.)): | ||
- | {{: | + | {{: |
After these general settings the programm continues in a rather standard manner with a section collecting various declarations of sets and parameters. Among these are the general sets of CAPRI (sets.gms), and the sets specific to the market model (arm_sets.gms) because the purpose of the task is to compile the data needed for the market model at the global level of CAPRI: | After these general settings the programm continues in a rather standard manner with a section collecting various declarations of sets and parameters. Among these are the general sets of CAPRI (sets.gms), and the sets specific to the market model (arm_sets.gms) because the purpose of the task is to compile the data needed for the market model at the global level of CAPRI: | ||
- | {{: | + | {{: |
The most important data source for task “Build global database” is FAOstat which involves a fairly long file (FAO_codes_new.gms) with sets and cross-sets to map from FAO regions, items, and products into the CAPRI world (defined by the code system in the annex). This serves to map some key data from FAO compiled in the previous task: population (fao_population.gms), | The most important data source for task “Build global database” is FAOstat which involves a fairly long file (FAO_codes_new.gms) with sets and cross-sets to map from FAO regions, items, and products into the CAPRI world (defined by the code system in the annex). This serves to map some key data from FAO compiled in the previous task: population (fao_population.gms), | ||
- | {{: | + | {{: |
The second most important group of data, both historical as well as projections, | The second most important group of data, both historical as well as projections, | ||
- | {{:code_p_104_3.png? | + | {{:code_p104_3.png?600}} |
* The first $include file (load_%aglink%_new.gms((A string like %textname% is a placeholder in GAMS code for some other text to be substituted for %textname% during the program execution. In this example it holds the name for the specific Aglink-Cosimo version that should be loaded.)) ) includes the relevant sets to handle the Aglink data, including the cross-sets to map to CAPRI. In addition it also merges a special data set on fish markets with other original Aglink data. | * The first $include file (load_%aglink%_new.gms((A string like %textname% is a placeholder in GAMS code for some other text to be substituted for %textname% during the program execution. In this example it holds the name for the specific Aglink-Cosimo version that should be loaded.)) ) includes the relevant sets to handle the Aglink data, including the cross-sets to map to CAPRI. In addition it also merges a special data set on fish markets with other original Aglink data. | ||
Line 2103: | Line 2616: | ||
The next three $include files cover additional macroeconomic data from UNstats (load_gdp_unstats_new.gms), | The next three $include files cover additional macroeconomic data from UNstats (load_gdp_unstats_new.gms), | ||
- | {{: | + | {{: |
It may be seen that “create_longrun_info.gms” is active or not depending on a setting from the GUI or a batch file. Similar to the code processing Aglink information it includes sets and mappings to handle the GLOBIOM information. Another similarity with the Aglink related files is that this code basically needs annual adjustments, | It may be seen that “create_longrun_info.gms” is active or not depending on a setting from the GUI or a batch file. Similar to the code processing Aglink information it includes sets and mappings to handle the GLOBIOM information. Another similarity with the Aglink related files is that this code basically needs annual adjustments, | ||
Line 2109: | Line 2622: | ||
The CAPRI market model is spatial and therefore requires data on bilateral trade flows. These are covered in two include files, the first one dealing with the special case of biofuel trade flows, the second one with the general case. \\ | The CAPRI market model is spatial and therefore requires data on bilateral trade flows. These are covered in two include files, the first one dealing with the special case of biofuel trade flows, the second one with the general case. \\ | ||
- | {{: | + | {{: |
Biofuel trade requires a special treatment again because FAOstat does not cover these. Instead, bilateral trade flows are constructed using total exports and imports from AGLINK and trade data from COMEXT, USDA and FO-Licht. By contrast the data for the trade matrix for other commodities is from FAOstat. | Biofuel trade requires a special treatment again because FAOstat does not cover these. Instead, bilateral trade flows are constructed using total exports and imports from AGLINK and trade data from COMEXT, USDA and FO-Licht. By contrast the data for the trade matrix for other commodities is from FAOstat. | ||
Line 2119: | Line 2632: | ||
The second is a transport cost matrix estimation using the original FAOstat trade matrix (so before gap filling and consolidation) and distance related information from CEPII. Together with price information the transport costs are estimated to provide a link between CIF and FOB prices for bilateral tradeflows. | The second is a transport cost matrix estimation using the original FAOstat trade matrix (so before gap filling and consolidation) and distance related information from CEPII. Together with price information the transport costs are estimated to provide a link between CIF and FOB prices for bilateral tradeflows. | ||
- | {{: | + | {{: |
The next $include file extends the Aglink-Cosimo projections to 2030, if needed, with a trend estimation involving a number of pragmatic modifications (such as the trend line passing trough the last observation). Then the the growth factors computed previously or the default trends are used to estimate a medium term outlook projections for global market balances, prices or GDP. These projections do however not include any consictency checks on closed market balances or similar properties. This is achieved in the baseline calibration only. | The next $include file extends the Aglink-Cosimo projections to 2030, if needed, with a trend estimation involving a number of pragmatic modifications (such as the trend line passing trough the last observation). Then the the growth factors computed previously or the default trends are used to estimate a medium term outlook projections for global market balances, prices or GDP. These projections do however not include any consictency checks on closed market balances or similar properties. This is achieved in the baseline calibration only. | ||
- | {{: | + | {{: |
Finally, data on trade policy variables such as applied and scheduled tariffs, tariff rate quotas or bilateral trade agreements are collected from the Agricultural Market Access Database (AMAD, obsolete current version) or from the MacMaps database (%macMap%)((See GAMS Documentation on The GAMS Call and Command Line Parameters (https:// | Finally, data on trade policy variables such as applied and scheduled tariffs, tariff rate quotas or bilateral trade agreements are collected from the Agricultural Market Access Database (AMAD, obsolete current version) or from the MacMaps database (%macMap%)((See GAMS Documentation on The GAMS Call and Command Line Parameters (https:// | ||
Line 2129: | Line 2642: | ||
The very last include file is probably also the least important one: FAPRI projections had a more important role several years ago, are not updated anymore and presumable affect less than a dozen numbers (if any at all) in the global database compiled in this task: | The very last include file is probably also the least important one: FAPRI projections had a more important role several years ago, are not updated anymore and presumable affect less than a dozen numbers (if any at all) in the global database compiled in this task: | ||
- | {{: | + | {{: |
=====Policy data===== | =====Policy data===== | ||
Line 2154: | Line 2667: | ||
===Tariffs and Tariff Rate Quotas=== | ===Tariffs and Tariff Rate Quotas=== | ||
- | Data on trade policy instruments other than tariffs (Tariff Rate Quotas, export subsidies, entry price system and flexible levies) enter CAPRI directly in the market model calibration workstep. Note that the ad valorem equivalent tariff rates in MacMap already include an estimated equivalent tariff rate for TRQs. Nevertheless, | + | Data on trade policy instruments other than tariffs (Tariff Rate Quotas, export subsidies, entry price system and flexible levies) enter CAPRI directly in the market model calibration workstep. Note that the ad valorem equivalent tariff rates in MacMap already include an estimated equivalent tariff rate for TRQs. Nevertheless, |
The TRQ system of the EU is included in great detail, based on DG AGRI.information. Data on TRQ orders are aggregated to the geographical and commodity definitions of CAPRI in dat/ | The TRQ system of the EU is included in great detail, based on DG AGRI.information. Data on TRQ orders are aggregated to the geographical and commodity definitions of CAPRI in dat/ |
the_capri_data_base.txt · Last modified: 2024/07/05 06:24 by massfeller