Differences

This shows you the differences between two versions of the page.

--- the_capri_data_base [2021/12/13 11:11] – [Data Import] himics
+++ the_capri_data_base [2021/12/14 12:27] – [Annex: Testing procedure and checking intermediate steps in COCO (biofuels)] himics
@@ Line 1032: / Line 1032: @@
 After this last include file the completions in module COCO2 are finished and the main output file (coco2_output.gdx) is unloaded. This file is loaded in subsequent modules (main use in CAPREG, but also in CAPTRD for nowcasting and in CAPMOD for update of LULUCF coefficients).
+====Annex: Code lists for the COCO database====
+This section includes detailed code lists, which are in use in the COCO database.
+**Table: Codes used for storing the original REGIO tables in the database and their description (rows)**
+^ Codes used in CAPRI REGIO tables ^ Original REGIO description ^
+| TOTL | Territorial area |
+| FORE | Forest land |
+| AGRI | Utilized agricultural area |
+| GARD | Private gardens |
+| GRAS | Permanent grassland |
+| PERM | Permanent crops |
+| VINE | Vineyards |
+| OLIV | Olive plantations |
+| ARAB | Arable land |
+| GREF | Green fodder on arable land |
+| CERE | Cereals (including rice) |
+| WHEA | Soft and durum wheat and spelt |
+| BARL | Barley |
+| MAIZ | Grain maize |
+| RICE | Rice |
+| POTA | Potatoes |
+| SUGA | Sugar beet |
+| OILS | Oilseeds (total) |
+| RAPE | Rape |
+| SUNF | Sunflower |
+| TOBA | Tobacco |
+| MAIF | Fodder maize |
+| CATT | Cattle (total) |
+| COWT | Cows (total) |
+| DCOW | Dairy cows |
+| CALV | Other cows |
+| CAT1 | Total cattle under one year |
+| CALF | Slaughter calves |
+| CABM | Male breeding calves (\&lt;1 year) |
+| CABF | Female breeding calves (\&lt;1 year) |
+| BUL2 | Male cattle (1-2 years) |
+| H2SL | Slaughter heifers (1-2 years) |
+| H2BR | Female cattle (1-2 years) |
+| BUL3 | Male cattle (2 years and above) |
+| H3SL | Slaughter heifers (2 years and above) |
+| H3BR | Breeding heifers |
+| BUFF | Total buffaloes |
+| PIGS | Total pigs (total) |
+| PIG1 | Piglets under 20 kg |
+| PIG2 | Piglets under 50 kg and over 20 kg |
+| PIG3 | Fattening pigs over 50 kg |
+| BOAR | Breeding boars |
+| SOW2 | Total breeding sows |
+| SOW1 | Sows having farrowed |
+| GILT | Gilts having farrowed for the first time |
+| SOWM | Maiden sows |
+| GILM | Maiden gilts |
+| SHEP | Sheep total) |
+| GOAT | Goats (total) |
+| EUQI | Equidae (total) |
+| POUL | Poultry (total) |
+| OUTP | Final production |
+| CROP | Total crops production |
+| DWHE | Durum wheat |
+| PULS | Pulses |
+| ROOT | Roots and tubers |
+| INDU | Industrial crops |
+| TEXT | Textile fibre plants |
+| HOPS | Hops |
+| VEGE | Fresh vegetables |
+| TOMA | Tomatoes |
+| CAUL | Cauliflowers |
+| FRUI | Fresh fruit |
+| APPL | Apples |
+| PEAR | Pears |
+| PEAC | Peaches |
+| CITR | Citrus fruit (total) |
+| ORAN | Oranges |
+| LEMN | Lemons |
+| MAND | Mandarins |
+| GRAP | Table grapes |
+| WINE | Wine |
+| TABO | Table olives |
+| OLIO | Olive oil |
+| NURS | Nursery plants |
+| FLOW | Flowers and ornamental plants |
+| OCRO | Other crops |
+| ANIT | Total animal production |
+| ANIM | Animal |
+| SHGO | Sheep and goats |
+| ANIP | Animal products |
+| MILK | Milk |
+| EGGS | Eggs |
+| INPU | Intermediate consumption (total) |
+| FEED | Animal feeding stuffs |
+| FDGR | Animal compounds for grazing livestock |
+| FDPI | Animal compounds for pigs |
+| FDPO | Animal compounds for poultry |
+| FODD | Straight feeding stuffs |
+| FERT | Fertilizers and enrichments |
+| ENER | Energy and lubricants |
+| INPO | Other inputs |
+| GVAM | Gross value added at market prices |
+| SUBS | Subsidies |
+| TAXS | Taxes linked to production (including VAT balance) |
+| GVAF | Gross value added at factor costs |
+| DEPM | Depreciation |
+| LABO | Compensation and social security contributions of employees |
+| RENT | Rent and other payments |
+| INTE | Interests |
+| GFCF | Total of gross fixed capital formation |
+| BUIL | Buildings and other structures |
+| MACH | Transport equipment and machinery |
+| GFCO | Other gross fixed capital formation |
+====Annex: Detailed description of Eurostat data processing in COCO (coco1_eurostat.gms)====
+The program starts by importing pre-processed data from Eurostat. The pre-processing includes simple data selection routines and also manual checks. The Eurostat domains are processed one by one, and the corrections are done for each Member State ((Eurostat offers data for Belgium and Luxembourg separately, whereas the database combines both countries to the model region "BL000" (Belgium and Luxembourg). The key reason is that Eurostat offers data mainly for the aggregate Belgium and Luxembourg up to the year 1999, especially for all market balances. Furthermore, Luxembourg has a rather small agricultural sector (2004 total output was about EUR 250 million) with some similarities to Belgium.))
+Below we discuss the specific data-processing tasks related to Eurostat table groups.
+The first Eurostat Table Group is “p_AgriProd” covering market balances and activity levels.
+//Corrections and complements for all MS://
+  * The following  data are not anymore available form Eurostat, starting with the 2010 data extractionBeginning with Eurostat selection 2010 some data are missing from the Eurostat website:
+    * DWH1, RAP1, POT1, POT2, ROO1 and ROO2 are not longer supported
+    * data for slaughter heads and slaughter tons for calves are only available for recent years
+    * deliveries to dairy of RMLK missing for earlier years in selection starting with February 2018
+For an Interim solution, data for the missing data points are collected from an earlier Eurostat selection (March 2010).
+  * UNFCCC data is included, here sheep and goats population, to prolong data of some countries where Eurostat data collection stopped 2008/2009.
+  * Recent dairy sector data from Eurostat via DG supplements the ordinary dairy data downloaded from the website of Eurostat.
+  * Sugar trade data from the market balances of Eurostat is extended with Comext (Eurostat) data.
+  * For the milk products WMIO, SMIP, FRMI and COCM some market balance positionpositions are corrected: “industrial use” is added to “feed on market and “processing” is added to “human consumption.
+  * COCO code "FRUI" is aggregated from auxiliary data for fruit trees, plus soft fruits, plus strawberries.
+  * All activities for the aggregate ILAM are added up from SHEP and GOAT.
+  * The units for wine balance sheets are converted from 1000hl to 10000hl=1000000l
+  * A rice milled equivalent balance without paddy rice (separate product) is constructed.
+  * Survey data on buffaloes are used to increase the bovine stock data to cover the whole cattle herd.
+//Corrections and complements for specific MS://
+Due to years of database updates, a number of corrections on input data are carried out. For special cases in some MS, data are read in from additional data sources:
+  * Belgium-Luxemburg: trade for potatoes (Eurostat: EU trade since 1988 by HS2-HS4 [DS-016894])
+  * France: market balances for cereal products (Agreste, Direction générale des douanes et droits indirects (DGDDI))
+  * Denmark: market balances for some cereal products (StatBank Denmark)
+  * Finland: market balances for some cereal products (Natural Resources Institute Finland, Balance sheet for food commodities)
+  * Germany: activity levels for textile crops (BMELF)
+  * Ireland: trade for citrus fruits and some milk products (Eurostat: EU trade since 1988 by HS2-HS4 [DS-016894]) and activity levels for grass land (StatBank Ireland)
+  * Austria: production of cow milk, fruit products and potatoes (Statistisches Amt Österreich)
+  * Czechia: trade of life animals (Eurostat: EU trade since 1988 by CN8 [DS-016890])
+  * Lithuania: human consumption cereal products (calculated from data from statistical yearbook 2018)
+  * Slovenia: slaughtering (SiStat Slovenia)
+  * Romania: data for the meat and in the milk sectors (Romanian experts)
+  * Trade data for sugar are collected from Eurostat COMEXT data.
+The remaining domains/table groups only require a few case-by-case corrections:
+  * The second Eurostat Table Group is “p_ExchRate” covering exchange rates
+  * The third Eurostat Table Group is “p_EcoAct” covering the economic accounts for agriculture.
+  * The fourth Eurostat Table Group is “p_AgriPri” covering agricultural producer prices.
+====Annex: Testing procedure and checking intermediate steps in COCO (biofuels)====
+The COCO module produces various reporting files on the intermediate data processing steps. These files can be used to trace back potential errors in the COCO database to their origin. These debugging files also contain meta-information on the input data and settings used for producing the COCO database.
+The following example is a walk-through on the typical data processing steps, covring biofuels data preparation in France.
+The reporting file 'C:\Users\himics\c\dairybiof\output\results\coco\biof_data_with_prep\chk_biof_data_with_prep_FR000000.gdx' reports on the data preparation for biofuels for France (FR000) in COCO1. The file includes the set ‘meta_prepare_biofuel_data’, with meta-information on the recent coco1 run (e.g. creation date of file, GAMS version used).
+In addition to the CAPRI activity codes (see Annex on code lists above), the following codes are also used:
+| bioECere | Ethanol processed from cereals |
+| bioESuga | Ethanol processed from sugar beets |
+| bioETwin | Ethanol processed from wine |
+| bioEFrui | Ethanol processed from fruits |
+| bioEOcro | Ethanol processed from other agricultural crops |
+| bioEExog | Ethanol processed from crops not explicit in biofuel modelling (fruits, potatoes, other crops) |
+| bioARES | Biofuels processed from crops residues |
+| bioORES | Biofuels processed from forest residues and waste material (municipal waste, waste oil, other waste) |
+| SECG | Biofuel quantities from second generation |
+| MAPRagr | Ethanol production from agricultural sources |
+| EloBio | Biofuel production and demand data from DG Energy project EloBio |
+| DG\_Agri | Ethanol data from DGAgri website and supplementary files |
+| ProdCom | Eurostat: PRODCOM ANNUAL SOLD (NACE Rev. 2.) [DS-066341] |
+| EIA | Independent Statistics &amp; Analysis, US Energy Information Administration |
+| comext | Eurostat: Comext |
+| Energy\_bal | Eurostat: Supply, transformation, consumption - renewable energies - annual data [nrg\_107a] |
+| Energy\_dem | Eurostat: Supply, transformation, consumption - renewable energies - annual data [nrg\_102a, nrg\_1073a] |
+| final | results of the calculations |
+| ODOM | other domestic use (activity from biostock calculations |
+| APRagr | Ethanol production from agricultural sources |
+| INDt | Sum of model results for BIOF and INDM |
+| BIOi, INDi, DOMi | intermediate activities to save data from model initialisation for later documentation. |
+====Annex: Testing procedure and checking intermediate steps in COCO (dairy)====
+The following three examples show how to use the intermediate reporting files to trace the data preparation steps. Screenshots demonstrate the arrangement of the reporting parameters by using the CAPRI Graphical User Interface.
 ===== The Regionalised Data Base (CAPREG) =====
 ==== Data requirements and sources at the regional level ====