the_complete_and_consistent_data_base_coco_for_the_national_scale
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
the_complete_and_consistent_data_base_coco_for_the_national_scale [2020/02/13 09:39] – [COCO2: Estimation procedure] matsz | the_complete_and_consistent_data_base_coco_for_the_national_scale [2020/03/27 07:02] – [COCO1: Overlay from various sources] matsz | ||
---|---|---|---|
Line 295: | Line 295: | ||
** Table 5: | ** Table 5: | ||
+ | |||
+ | ^ Conversion coefficients (t/t) ^^Ethanol ^Byproducts ^ | ||
+ | |**Grains**|Wheat| 0.274| 0.266 DDGS| | ||
+ | |:::|Barley |0.247 |0.266 DDGS| | ||
+ | |:::|Oats |0.247 |0.266 DDGS| | ||
+ | |:::|Rye |0.247 |0.266 DDGS| | ||
+ | |:::|Corn (dry milling) |0.335 |0.292 DDGS| | ||
+ | |**Other** |Table Wine| 0.100| | | ||
+ | |**Sugar Crops**|Sugar |0.517| | | ||
+ | |:::|Sugar beets |0.079 |0.004 Vinasses*| | ||
+ | ^ ^^Biodiesel ^Byproducts^ | ||
+ | |**Veegetable oils** |Rape oil| 0.922 |0.100 Glycerine| | ||
+ | |:::|Soy oil |0.922 |0.100 Glycerine| | ||
+ | |::: | ||
+ | |:::|Palm oil |0.922 |0.100 Glycerine| \\*considered as molasses (1t vinasses = 0.1t molasses equivalent) depending on the reduced sugar content \\ Source: Own compilation base on AgLink database, PRIMES questionnaire and Szulczyk, K /2007) | ||
+ | |||
+ | Note: The beet coefficient has been increased in the meantime from 0.079 to 0.086. | ||
+ | |||
+ | //Fuel prices and taxes// | ||
+ | |||
+ | For a specification of processing-, | ||
+ | |||
+ | ===Assigning data to database array=== | ||
+ | |||
+ | So far data processing has focussed on the key Eurostat Table Group “apro” (collected on parameter p_agriProd). The next parts of COCO will collect data from other sources, including the other two Table Groups for prices and Economic accounts (“apri”, | ||
+ | |||
+ | {{:: | ||
+ | |||
+ | **Include file ‘// | ||
+ | |||
+ | This file collects expert data for specific countries that receive priority over all other data sources in the initialisaiton. | ||
+ | |||
+ | **Include file ‘// | ||
+ | |||
+ | This sub-module assigns the areas, crop production data and most market balance positions from Eurostat’s | ||
+ | |||
+ | After this special case the crop areas from Eurostat' | ||
+ | |||
+ | Inappropriate aggregation (ignoring gaps in the component series) has been frequently observed in past experiences with Eurostat data such that aggregates are added up, if possible, from any given sub-components. This principle applies to " | ||
+ | |||
+ | In terms of gross production (GROF) it has to be mentioned that preference is given to the market balance information “USAP” over the production statistics “GROF”), | ||
+ | |||
+ | For products with market balance \(DATA(GROF, | ||
+ | Remaining products \(DATA(GROF, | ||
+ | |||
+ | Some special assingments handle SEDF and LOSF for cereals and the residual calculation of production of " | ||
+ | |||
+ | More important is a procedure to ensure a complete initialisation of fodder production quantities, an area with widespread gaps in the raw data. This procedure estimates fodder yields (of " | ||
+ | |||
+ | Contrary to the program name, //all// balance positions for crops //and animals//, except milk positions, are assigned to the " | ||
+ | |||
+ | In several cases upper or lower limits are assigned for qunatities and areas where it turned out that missing data are often completed in the optimisation part of COCO in an unsatisfactory way. The empirical basis for these limits is diverse. It may rest on production statistics (if production is given there but missing in the market balances), on sugar quotas for the sugar beet sector, or in some cases (fruits, vegetables) on a moving average over given observations. | ||
+ | |||
+ | **Include file ‘// | ||
+ | |||
+ | This file assigns the data for dairy products and raw milk from Eurostat' | ||
+ | |||
+ | Gross production of raw milk is usually given from the farm balance data (COMI = CMLK, cow milk + BMLK, buffalo milk. SGMI = EMLK, ewes milk + GMLK goats milk). | ||
+ | |||
+ | Gaps are more frequent for deliveries to dairies (" | ||
+ | |||
+ | Whereas production data and deliveries to dairies may be distinguished into “COMI” and “SGMI”, the dairy statistics on derived products obtained or associated market balances do not permit such distinction. As a consequence, | ||
+ | |||
+ | The content of milk products is initialised using two types of information: | ||
+ | |||
+ | **Include file ‘// | ||
+ | |||
+ | Assigning herd size, process length, activity level, yield and production data often requires significant reaggregations from the slaughtering statistics and therefore explanations in this documentation: | ||
+ | |||
+ | The first best source for tons of slaughtered meat of the main animal categories (SLGT.IPIG, ILAM, ICAT and ICHI) is the usable production (USAP) from the balance sheets because this is likely to be consistent with market balances. As a second best source we use the slaughtering statistics, but with a correction factor. Export and imports of live animals expressed in carcass weight are partly taken from the slaughtering statistics or from the balance sheets, depending on availability. It is useful to remember that total production of meats in heads (e.g. " | ||
+ | |||
+ | Herd size data are initialised based on the data prepared in ‘// | ||
+ | |||
+ | For cattle other than cows the activity level definition is more complex. In the case of heifers and bulls for fattening, the activity level equals the number of slaughtered heads plus net exports of live animals. If slaughtered heads of heifers and bulls are unavailable, | ||
+ | |||
+ | In the same manner the number of heifers needed as input (GROF.IHEI) for each year is equal to the activity levels of heifers for raising and heifers for fattening. The number of female calves raised (activity level) in the current year is equals the number of heifers used as inputs in the following year. Similarly the number of young bulls raised equals next year’s production of adult male cattle in heads. In countries with complete statistical data there are only two activity levels that cannot be fully inferred from statistical data alone: As the statistics do not distinguish slaughterings and trade of male and female calves we are using a male share of 51% to estimate the split of male and female calves. This also permist to calculate the total number of calves of each sex needed as input for each year as calves for raising plus calves for fattening and correspondingly the output coefficient of cows. Conversely the output coefficients of calves in terms of beef may be calculated from statistical data on slaughtered calves in tons and heads. | ||
+ | |||
+ | Herd size data usually may be mapped exactly to particular cattle categories in the CAPRI data base, including the distinction of heifers for raising and for fattening. The only exception is the distinction of the herd size of male and female calves which is assigned according to the estimated split in the related activity levels. Having assigned both the herd size as well as activity levels permits to assign: average process length in days = activity level / herd size * 365. The average process length in turn is related to the daily growth of animals according to another accounting identity: final (live) weight = beginning (live) weight + daily growth * (process length – empty days). This accounting identity will be imposed in the COCO1 estimation procedure, but module coco1_anim assigns bounds (parameters UppLim and LowLim) for the process length such that the implied daily growth values remain in a reasonable range. For heifers there is also an upper bound for the process length for statistical reasons: female animals older than 36 months are classified as “cows”, whether they have calved or not. | ||
+ | |||
+ | Activity levels and slaughter weights for animal types other than cattle are more straightforward to obtain. The herd size of fattenened pigs beyond 20 kg, of piglets up to 20 kg and sows (+ boars) is the average number according to the four possible annual counting (April, May/June, August and December). The number of fattened pigs (flow of animals) equals total slaughtered pigs minus slaughtered sows. The output coefficient (piglets) per sow equals the number of slaughtered pigs plus the increase in the sows herd size. The input coefficient is an estimate of sows slaughterings per sow (inferred from stock data on young sows and the stock change of all sows). The production of pork from pigs for fattening is calculated from total meat production less the pork from sows, assuming that a sow produces 120 kg of meat. | ||
+ | |||
+ | Two particularities in the pig sector are worth mentioning. The first is that as of 2011 the COCO database includes the herd size of piglets < 20kg (on code PIGL00.HERD) even though there is no explicit activity level “raising of piglets”. Instead the piglets raised are one of the outputs of activity sows with total production of piglets given on code GROF.YPIG. Accordingly we cannot store the process length for raising of piglets in a column for “raising of piglets” but introduce a new code “PIGF.YDAYS” such that in the completed data base we find the relationship PIGF.YDAYS = GROF.YPIG / PIGL00.HERD * 365. Including the piglets turned out useful because it permits to make use of statistical data on the total pigs population which is sometimes available even though pig slaughterings in heads are missing. | ||
+ | |||
+ | The second pig sector particularity relates to the requirement functions for pigs, stored in the form of a table (// | ||
+ | |||
+ | In the poultry sector we have herd size data for chicken broilers, turkeys, ducks, and geese (yearly average, mainly from FAO) and hens from Eurostat (average of this and last year’s December counting). The first four give the total herd size of poultry for fattening whereas the herd size of hens also equals the activity level. The output coefficient for eggs relies on usable production from the balance sheets divided by the herd size of hens. A replacement rate of 80% is assumed for laying hens. The activity level of poultry fattening is the difference of total produced poultry heads minus slaughtered hens. The output coefficients and production in terms of meat are straightforward to calculate from here. With activity level and aggregate herd size of poultry for fattening being defined it is possible to calculate the implied process length. The information on the shares of chicken broilers, turkeys, ducks, and geese is used to specify technical bounds for the daily growth and process length. In addition the technical literature also permitted to specify typical empty days for cleaning of stables (or seasonality in the case of geese and ducks). The differentiation of poultry for fattening is only maintained temporarily in COCO1 because it helped to use statistical information for the specification of some technical coefficients that strongly depend on the shares of turkeys. Subsequent CAPRI modules (like CAPREG) will only use the COCO results for the aggregate poultry fattening activity (POUF). | ||
+ | |||
+ | The herd size data for sheep and goats are assigned in the same way as for cattle. The herd size of sheep and goats for milk is at the same time the activity level. The number of slaughtered lambs (sheep and goats) is the total slaughtering number (including net exports of young animals) minus the slaughtering of adults. This estimate for slaughtered lambs in heads also defines the activity level of sheep and goats for fattening. The total output in tons set equal to the meat production. A particularity in the sheep and goat sector is the strong seasonality in some countries. Empty days are specified based on the share of the December counting (sheep in continuous systems) to the May-June counting (sheep in seasonal + continuous systems). These enter the specification of bounds for the process length in sheep and goats fattening. | ||
+ | |||
+ | **Include file ‘//coco1 assign_AgriPri.gms// | ||
+ | |||
+ | Before assigning the prices from p_agriPri to the tareget parameter data 3 issues are addressed: | ||
+ | |||
+ | * Price differences in the original series between MS suggested that not all series have been already expressed "per nutrient" | ||
+ | * Prices for dairy products CHES and COCM need aggregation from more specific series | ||
+ | * Outliers are identified according to limits for plausible differences to the EU average | ||
+ | |||
+ | **Include file ‘//coco1_ candi_EcoAct.gms// | ||
+ | |||
+ | Except for Macedonia, which reports EAA data to Eurostat, all other candidate countries receive an EAA initalisation from previously assigned GROF times PRIC. Input positions are assigned based on shares borrowed from an average across selected EU MS. | ||
+ | |||
+ | **Include file ‘// | ||
+ | |||
+ | In this file EAA data from Eurostat are assigned from parameter p_ecoAct to data(.), including unit values. For a number of aggregates special assignments are needed to obtain monetary values matching with the aggregates used elsewhere in COCO. | ||
+ | |||
+ | Unit values at producer price are preferably calculated as a quotient from the value at producer price and the quantity as selected from the EAA statistics. However some checks are used to discard grossly implausible (outlier) unit values. | ||
+ | |||
+ | To serve as a fall back option for the EAA unit values, the previously assigned prices from the p_agriPri parameter are corrected to acknowledge the typical differences between producer prices (UVAP) and selling prices (PRIC). Finally, if price indices are still missing for single items, those from product groups are used. | ||
+ | |||
+ | Prices for energy positions heating gas EGAS and fuel EFUL may be used to infer quantity variables in CAPREG from value information. A special section takes care for completeness. | ||
+ | |||
+ | Finally production of non-physical items from the EAA (some outputs like NURS, FLOW and inputs other than heating gas EGAS and fuel EFUL) may be calculated by the quotient of EAA value and a price index. As we will also express the output “quantity” for heterogenous items “other industrial crops” (OIND), “other crops” (OCRO) and “other animal products” (OANI) in values at constant prices (currently 2005), the complete list of non-physical items with quantity information given as values in constant prices is (using the codes from the end of this documentation): | ||
+ | |||
+ | Outputs: NURS, | ||
+ | |||
+ | Inputs: IPHA, | ||
+ | |||
+ | With coco1_eaa.gms passed, the presumably best raw data are collected on the central parameter data(.), but a few additional completions are possible to inprove the internal consistency of the initialisation before proceeding to the main consolidation steps: | ||
+ | |||
+ | {{:: | ||
+ | |||
+ | **Include file // | ||
+ | |||
+ | This file calculates residuals from the given data for aggregates and sub-positions for crops. The residual activity level and market balance position is defined as a difference between the group level and the sum of individual crops. This calculation is not carried out if there are gaps in some components or if the total is smaller than the sum of given components. | ||
+ | |||
+ | **Include file // | ||
+ | |||
+ | Yields are evidently calculated for each crop activity by dividing the gross production by the production level for this activity. However, this sub-module also applies a Hodrick-Prescott (HP) filter to smooth out problems with yields from activities with small production areas. | ||
+ | |||
+ | **Include file // | ||
+ | |||
+ | In most countries grass is the most important ‘crop’ in terms of area use yet, often the data on grass areas and production are one of the weakest parts of crop statistics. When relying solely on statistical data, the COCO database frequently showed unbelievable grass yields in some MS. This sub-module assigns grass yields, based on expert knowledge, to be used as priori information together with statistical data in part 2 of the COCO routine. The key information is expert data((These were estimates worked out in September 2006 by Oene Oenema and Gerard Velthof from Alterra, Wageningen, in the context of a service contract for DG-ENV (Integrated measures in agriculture to reduce ammonia emissions, No 070501/ | ||
+ | |||
+ | **Include file ‘// | ||
+ | |||
+ | This file allows to process information from various sources on the same item, in particular areas for various land use items (“LEVL”). In order to handle the different sources, new rows are defined, indicating from which source the information on land use area is coming which is typically only offered for a selected years or a limited period: | ||
+ | |||
+ | * **LEVAgriProd** - Eurostat national land use data (Eurostat Table: “apro_cpp_luse”, | ||
+ | * **LEVCLC** - Land use levels derived from Corine Land Cover (CLC) using a transformation matrix to LUCAS in two steps | ||
+ | * Original Corine Land Cover (44 classes, aggregated to the NUTS2 level((Data for some countries and years affected by evident problems have been removed. For example the 2006 CLC data only covered parts of Greece, hence are no usable to calculate totals at the MS level.)) obtained from JRC, Ispra for 1990, 2000, 2006, 2012. To link the Corine information to the CAPRI land use classes we used as an interim step so-called contingency tables from CLC to LUCAS categories provided by JRC Ispra at NUTS2 level. This allows to map the Corine classes (like complex cultivation patterns – “complexCultiv”) to the //most probable// land cover class from the LUCAS survey (in the example “complexCultiv” -> annual crops) which may be aggregated then to the CAPRI land use aggregates (annual crops LUCAS -> arable crops, CAPRI code ARAC). However, while this mapping to the //“most probable”// | ||
+ | * To acknowledge that the Corine Classes may be mapped to several LUCAS categories we multiplied them with the “profiles”, | ||
+ | * **LEVRegio** - Eurostat regional land use data (Eurostat Table: “agr_r_landuse”, | ||
+ | * **LEVFAO** - Land use data from the resource FAOSTAT domain((See [[http:// | ||
+ | * **LEVLucas** – directly using the LUCAS data is an option that has been considered but not implemented in CAPRI so this code is not used at the moment. | ||
+ | * **LEVLandCov** - Eurostat land cover data for 2009, 2012, 2015 at the MS level. Agricultural land is only distinguished into cropland CROP and grassland GRAS, but 5 nonagricultural areas are neatly aggregating up to the total country (Artificial ARTIF, shrubland (considered similar to “other wooded land” OWL), bare land & wetlands (mapped to “other sparcely vegetated or bare OSPA) and waters WATER. | ||
+ | * **LEVEnvio** - Eurostat land cover data from the environment section (Table “env_la_luc1”((Apparently these data are currently under revision because they are not accessible on the Eurostat website anymore since about June 2012. However they are still accessible (in July 2012) via [[http:// | ||
+ | * **LEVMcpfe** – Data from the Ministerial Conference on the Protection of Forests in Europe C&I database for quantitative indicators. This gives validated data on the forest sector (forest land FORE, other wooded land OWL) and some non forestry data (inland waters INLW, total country area ARTO), but data were only given for 1990, 2000, 2005, 2010, 2015. | ||
+ | * **LEVFSS** - Eurostat farm structure survey data (Table “ef_lu_ovcropaa" | ||
+ | * **LEVcrf** – The UNFCCC common reporting format (CRF) data (1990-2016), | ||
+ | |||
+ | These sources each provide information on some “land use classes” (Table 7 of Annex) at least. | ||
+ | |||
+ | **Include file ‘// | ||
+ | |||
+ | This file includes some final checks and adjustments before moving on to the optimisation part of coco. | ||
+ | * For seed quantities technical limits for reasonable seed use per ha are imposed. | ||
+ | * For all non crop products producer prices are assigned from the EAAP/UVAP positions or PRIC | ||
+ | * For all products with one of activity level, production or yield missing some correcting actions are taken. | ||
+ | * For FEDM, HCOM, SEDF and SEDM lower and upper limits are introduced to limit yearly changes in the subsequent estimation routines. | ||
+ | |||
====COCO1 Estimation procedure==== | ====COCO1 Estimation procedure==== | ||
Line 329: | Line 489: | ||
\end{split} | \end{split} | ||
\end{align} | \end{align} | ||
- | |||
\begin{align*} | \begin{align*} | ||
\begin{split} | \begin{split} | ||
- | &\text {s.t.}\\ | + | & \text {s.t.}\\ |
& | & | ||
&\text {Accounting identities defined on} y_{i,t}\\ | &\text {Accounting identities defined on} y_{i,t}\\ | ||
- | &\text {Identity of land use from different sources} | + | &\text {Identity of land use from different sources} |
\end{split} | \end{split} | ||
\end{align*} | \end{align*} | ||
- | |||
where //i// represents the index of the elements to estimate (crop production activities or groups, herd sizes etc.), //t// stands for the year, wgtx are weights attached to the different parts of the objective (\(wgt^{dat} = wgt^{hp} = 10, wgt^{ini} = 1, wgt^{up} = wgt^{lo} = 100)\), and | where //i// represents the index of the elements to estimate (crop production activities or groups, herd sizes etc.), //t// stands for the year, wgtx are weights attached to the different parts of the objective (\(wgt^{dat} = wgt^{hp} = 10, wgt^{ini} = 1, wgt^{up} = wgt^{lo} = 100)\), and | ||
Line 695: | Line 853: | ||
The previous snippet from coco2_feed gives an example that some sets (RS, R_RAGG) are assigned specifically to ensure functionality in different modules (here COCO2). | The previous snippet from coco2_feed gives an example that some sets (RS, R_RAGG) are assigned specifically to ensure functionality in different modules (here COCO2). | ||
- | As the name should signal file // | + | |
+ | As the name should signal file // | ||
Requirement functions are specified that determine: | Requirement functions are specified that determine: | ||
+ | |||
+ | * ENNE Net energy for ruminants as sum of | ||
+ | * NEL net energy for lactation (cows, ewes, goats) | ||
+ | * NEM net energy for maintenance (cows, calves, bulls, heifers, ewes, goats) | ||
+ | * NEA net energy for activity (cows, calves, bulls, heifers, ewes, goats) | ||
+ | * NEP net energy for pregnancy (cows) | ||
+ | * NEG net energy for growth (calves, bulls, heifers) | ||
+ | * ENMC Net energy chicken | ||
+ | * ENMP Net energy pigs | ||
+ | * CRPR crude protein (all categories) and LISI lysine aminoacid (sows, poultry) | ||
+ | * DRMA dry matter (all categories with min and max requirements) | ||
+ | * Various fiber measures (irrelevant for COCO2) | ||
+ | There are three main sources for these functions: | ||
+ | * IPCC 2006 guidelines for the estimation of emissions ([[http:// | ||
+ | * Kirchgessner Tierernährng, | ||
+ | * CAPRI working paper 97-12 ([[http:// | ||
+ | |||
+ | These functions are one the one hand quite complex. They are composed of various parts that finally give the requirements, | ||
+ | |||
+ | As a starting point, the daily growth from COCO is forced into the range defined in // | ||
+ | |||
+ | \begin{align} | ||
+ | \begin{split} | ||
+ | & | ||
+ | &= min [DAILY_{up}^{HEIF}, | ||
+ | \end{split} | ||
+ | \end{align} | ||
+ | |||
+ | The daily increase is then used to determine the process length (rearrangement of equation below with empty days EDAYS = 0) | ||
+ | |||
+ | \begin{align} | ||
+ | \begin{split} | ||
+ | & | ||
+ | &= min [DAYS_{up}^{HEIF}, | ||
+ | & \quad (BEEF_r^{HEIF}/ | ||
+ | \end{split} | ||
+ | \end{align} | ||
+ | |||
+ | The daily increase and process length may be conbined to estimate the mean live weight, | ||
+ | |||
+ | \begin{equation} | ||
+ | meanWgt_r^{HEIF}=startWgt_{HEIF}+\frac {dailyIncrease_r^{HEIF}\cdot fatngdays_r^{HEIF}} 2 | ||
+ | \end{equation} | ||
+ | |||
+ | which in turn is the last information to estimate energy requirements for maintenance according to the IPCC guidelines: | ||
+ | |||
+ | \begin{equation} | ||
+ | NEM_r^{HEIF}=(meanWgt_{HEIF})^{0.75}\cdot 0.322 \cdot fatngdays_r^{HEIF} | ||
+ | \end{equation} | ||
+ | |||
+ | Other energy requirements (for growth and activity) are calculated in a similar fashion as well as those for other animals. Important aspects to note are | ||
+ | |||
+ | * Fixed bounds for DAYS and DAILY ensure reasonable requirements, | ||
+ | * Regional coefficients are derived from the MS level information | ||
+ | |||
+ | **Include file // | ||
+ | |||
+ | With animal requirements specified the results of COCO1 for grass, other fodder and as a last resort cereals might be revised in terms of losses on farm to achieve an acceptable relationship of energy and protein requirements of total herds compared to the intake with feed. For gras and other fodder on arable land the contents may be adjusted in certain limits as well. The corrections do not eliminate the typical oversupply of nutrients compared to the requirements based on the literature, but they should give reasonable starting values for the feed allocation addressed in module CAPREG. | ||
+ | |||
+ | ===Compare COCO1 results with UNFCCC and compute correction factors in coco2_lulufc_carbon=== | ||
+ | |||
+ | In COCO1, an assignment of LULUCF effects (totals and per ha) has taken place, mostly relying on IPCC coefficients. These assignments are compared in coco2_lulucf_carbon with the reportings from EU MS to UNFCCC. For forestry and any transitions involving forestry, the standard IPCC reporting appears rather coarse, as it implies, for example, that management of forest land remaining forest has zero carbon effects. By contrast most EU countries report that there is still a considerable gain in biomass from forest management because the forests have not yet achieved a stable state (as implied by IPCC standard methodology). | ||
+ | |||
+ | To pick up the detailed knowledge of management practices, disturbances, | ||
+ | |||
+ | ===Complete prices for vegetable oil in coco2_oil_price=== | ||
+ | |||
+ | The EU prices for vegetable oils relevant for biofuel processing functions are assigned using prices from a USDA source. These assignments refer to prices at the wholesale level (relevant for the processing industry), not to consumer prices which have been determined previously. | ||
+ | |||
+ | After this last include file the completions in module COCO2 are finished and the main output file (coco2_output.gdx) is unloaded. This file is loaded in subsequent modules (main use in CAPREG, but also in CAPTRD for nowcasting and in CAPMOD for update of LULUCF coefficients). | ||
the_complete_and_consistent_data_base_coco_for_the_national_scale.txt · Last modified: 2022/11/07 10:23 by 127.0.0.1