Modelling of forecast errors in Data Assimilation for NWP "Uncertainty Analysis in Geophysical Inverse Problems" Lorentz Centre Workshops in Sciences series, Leiden, Netherlands. Nov 2011. Andrew Lorenc В© Crown copyright Met Office Content 0. Audience-specific orientation! 1. Historical context вЂ“ what is best for NWP? 2. Bayesian methods and error modelling 3. Adding non-Gaussianity В© Crown copyright Met Office Andrew Lorenc 2 В© Crown copyright Met Office Andrew Lorenc 2 В© Crown copyright Met Office Andrew Lorenc ECMWF Seminar 2011 3 Data Assimilation for Numerical Weather Prediction An assimilation cycle is an adaptive filter process: 1.Every 6~12 hours we make a forecast from the current best estimate, using best available computer forecast model 2.The DA process also provides an estimate of the error distribution of this forecast 3.We do a Bayesian combination of the prior from 1 & 2 with a batch of observations, to give a new best estimate valid 6~12 hours later. 4.We measure the properties of the actual forecast error, and at intervals adapt the algorithm used in 2 (and also in 1 & 3). Data Assimilation = The Scientific Method В© Crown copyright Met Office Andrew Lorenc ECMWF Seminar 2011 4 Historical Background: What has been important for getting the best NWP forecast?(over last 3 decades) NWP systems are improving by 1 day of predictive skill per decade. This has been due to: 1. Model improvements, especially resolution. 2. Careful use of forecast & observations, allowing for their information content and errors. 3. Advanced assimilation using forecast model: 4D-Var has been successful. 4. Better observations. В© Crown copyright Met Office Andrew Lorenc 5 В© Crown copyright Met Office Andrew Lorenc 5 Performance Improvements вЂњImproved by about a day per decadeвЂќ Met Office RMS surface pressure error over the N. Atlantic & W. Europe В© Crown copyright Met Office Andrew Lorenc 6 Peak Flops 60 Years of Met Office Computers 1.E+15 MooreвЂ™s Law 1.E+14 18month doubling time 1.E+13 1.E+12 1.E+11 1.E+10 1.E+09 1.E+08 1.E+07 1.E+06 1.E+05 KDF 9 1.E+04 1.E+03 Mercury LEO 1 1.E+02 1950 1960 1970 IBM Power -Phase 1&2 Cray T3E NEC SX6/8 Cray C90 Cray YMP Cyber 205 IBM 360 1980 1990 Year of First Use В© Crown copyright Met Office Andrew Lorenc 7 2000 2010 Historical Background: Continuing Improvement of a Complex System вЂў NWP improvements are due to a synergistic combination of improvements of forecast model, DA and observations. вЂў Each helps the other. вЂў Total NWP system is very large, complex and expensive. вЂў We cannot expect to understand it completely as a single entity. вЂў We cannot afford thorough testing of each improvement. вЂў Best to base each improvement on scientific insight and analysis of one component. вЂў With belief (checked by testing) that theoretically better parts will eventually give a better system. В© Crown copyright Met Office Andrew Lorenc 8 Implications for R&D strategy of causes of improvements to NWP 1. Model improvements, especially resolution. вЂў NWP models are very large (109 variables) and expensive to run. Nevertheless there are known shortcomings which could be addressed by making then larger and more expensive. вЂў DA improvements have to compete with the model for the use of computer upgrades. We cannot expect a large increase in the relative resources devoted to assimilation. вЂў Rely on linear DA methods to cope with 109 variables and millions of observations. Selected important nonlinearities can be added as corrections. В© Crown copyright Met Office Andrew Lorenc 9 Ratio of global computer costs: ra tio o f s u p e rc o m p u te r c o s ts : 11 dayвЂ™s DA (total incl. FC) / 1 dayвЂ™s forecast. d a y's a s s im ila tio n / 1 d a y fo re c a s t 100 Computer power increased by 1M in 30 years. Only 0.04% of the MooreвЂ™s Law increase over this time went into improved DA algorithms, rather than improved resolution! 10 31 20 4 D -V a r w ith o u te r_ lo o p s im p le 4 D -V a r on SX 8 8 3 D -V a r o n T3E 5 AC schem e 1 1985 1990 В© Crown copyright Met Office Andrew Lorenc 10 1995 2000 2005 2010 Operational NWP Models: 20th July 2011 Global пѓ�25km 70L пѓ�Hybrid 4DVAR вЂ“ 60km inner loop пѓ�60h forecast twice/day пѓ�144h forecast twice/day пѓ�+24member EPS at 60km 2x/day NAE пѓ�12km 70L пѓ�4DVAR вЂ“ 24km пѓ�60h forecast пѓ� 4 times per day пѓ� +24member EPS at 18km 2x/day UK-V (& UK-4) пѓ�1.5km 70L пѓ�3DVAR (3 hourly) пѓ�36h forecast Crown Copyright Office Met Office Global Regional Ensemble Prediction System = MOGREPS пѓ� 4 В©times per day2011. Source: Met Implications for R&D strategy of causes of improvements to NWP 2. Careful use of forecast & observations, allowing for their information content and errors. вЂў The forecast background summarises information from past observations. It is as accurate as most observations. вЂў Use incremental DA methods which correct the background based on a Bayesian updating with observed information. вЂў Need to understand the real information content of observations, e.g. satellite soundings observe radiances, not temperature profiles. В© Crown copyright Met Office Andrew Lorenc 12 Simplest possible Bayesian NWP analysis В© Crown copyright Met Office Andrew Lorenc 13 Simplest possible example вЂ“ 2 grid-points, 1 observation. Standard notation: Ide, K., Courtier, P., Ghil, M., and Lorenc, A.C. 1997: "Unified notation for data assimilation: Operational, Sequential and Variational" J. Met. Soc. Japan, Special issue "Data Assimilation in Meteorology and Oceanography: Theory and Practice." 75, No. 1B, 181вЂ”189 пѓ¦ x1 пѓ¶ x=пѓ§ пѓ· пѓ§x пѓ· пѓЁ 2пѓё Model is two grid points: y = пЂЁy o 1 observed value yo midway (but use notation for >1): o Can interpolate an estimate y of the observed value: y пЂЅ H пЂЁx пЂ© пЂЅ 1 2 x1 пЂ« 1 2 x 2 пЂЅ Hx пЂЅ пЂЁ 1 2 1 2 пѓ¦ x1 пѓ¶ пЂ©пѓ§пѓ§ пѓ·пѓ· пѓЁ x2 пѓё This example H is linear, so we can use matrix notation for fields as well as increments. В© Crown copyright Met Office Andrew Lorenc 14 пЂ© Bayes theorem in continuous form, to estimate a value x given an observation yo o o p(x | y ) = p( y | x)p(x) o p( y ) p(xп‚Ѕyo) p(x) p(yoп‚Ѕx) Can get p(yo) is the posterior distribution, is the prior distribution, is the likelihood function for x by integrating over all x: В© Crown copyright Met Office Andrew Lorenc 15 p( y ) = пѓІ p( y | x)p(x)dx o o background pdf We have prior estimate xb1 with error variance Vb: пЂЁ exp пЂЁ- p пЂЁ x 1 пЂ© = пЂЁ 2 пЃ° V b пЂ© exp - 1 2 p пЂЁ x 2 пЂ© = пЂЁ2пЃ° V b пЂ© 1 2 - 12 - 12 But errors in x1 and x2 are usually correlated пѓћ must use a multi-dimensional Gaussian: пЂЁ p( x 1 пѓ‡ x 2 ) = p пЂЁx пЂ© = (2 пЃ° ) | B | 2 пЂ© - 12 1 2 пЂ x пЂ© пЂ© пЂ x b 2 2 пЂ© V пЂ© V b b b x ~ N (x : x , B ) пѓ¦ 1 b T пЂ1 b пѓ¶ exp пѓ§ - пЂЁx - x пЂ© B пЂЁx - x пЂ©пѓ· пѓЁ 2 пѓё where B is the covariance matrix: В© Crown copyright Met Office Andrew Lorenc 16 пЂЁx пЂЁx b 2 1 пѓ¦ 1 B = V bпѓ§ пѓ§ пѓЁпЃ пЃпѓ¶ пѓ· пѓ· 1пѓё background pdf В© Crown copyright Met Office Andrew Lorenc 17 Observational errors Lorenc, A.C. 1986: "Analysis methods for numerical weather prediction." Quart. J. Roy. Met. Soc., 112, 1177-1194. o пЂЁ t y ~ N y ,E instrumental error пЂЁ пЂ© пЂЁ p y | y = пЂЁ 2пЃ° |E | пЂ© exp o - 12 1 2 пЂЁy o пЂ© пЂy пЂ© T E пЂ1 пЂЁy o пЂy пЂ©пЂ© Difference between the actual instrument and a hypothetical perfect instrument. For instance, for a wind observation from a radiosonde, the errors in tracking the balloon might lead to an instrumental error of about 1ms-1. y ~ N пЂЁ H пЂЁ x пЂ©,F пЂ© t error of representativeness пЂЁ t пЂ© пЂЁ2 ПЂ| F|пЂ© p t y |x = - 12 T пѓ¦ 1 t пѓ¶ пЂ1 t пЂЁ пЂ© пЂЁ пЂ© exp пѓ§ - y пЂ H x F y пЂ H x пѓ· пѓЁ 2 пѓё пЂЁ пЂ© пЂЁ Errors from H and model resolution in predicting the value from a perfect instrument. For instance, the error predicting a radiosonde wind from a model with grid-length 200km would be about 3ms-1, and a grid-length of 20km would reduce the error of representativeness to about 1ms-1. В© Crown copyright Met Office Andrew Lorenc 18 пЂ© Observational errors Lorenc, A.C. 1986: "Analysis methods for numerical weather prediction." Quart. J. Roy. Met. Soc., 112, 1177-1194. Observational error combines these 2 : y ~ N пЂЁ H пЂЁ x пЂ©,E + F пЂ© o пЂЁ o пЂ© пѓІ p пЂЁy | y пЂ© p пЂЁy | x пЂ©dy o p y |x = пЂЁ = пЂЁ 2 ПЂ| E + F |пЂ© exp - 12 1 2 t t t пЂЁy o пЂ H пЂЁx пЂ© пЂ© T Instrumental and representativeness errors are convolved to give a combined observational error: R=E+F. Note that we have assumed observations and background are unbiased. Treatment of biases is discussed in another lecture. We have also assumed that observational errors are uncorrelated with background errors. В© Crown copyright Met Office Andrew Lorenc 19 пЂ© пЂЁE + F пЂ©-1 пЂЁy o пЂ H пЂЁ x пЂ©пЂ© background pdf obs likelihood function В© Crown copyright Met Office Andrew Lorenc 20 Bayesian analysis equation пЂЁ p x| y o пЂ©= пЂЁ пЂ© p y | x p пЂЁx пЂ© o пЂЁ пЂ© p y o x ~ N пЂЁx , A пЂ© a Property of Gaussians that, if H is linearisable : where xa A and A are defined by: пЂ1 пЂЅB пЂ1 x пЂЅ x пЂ« AH R a b Remember that observational error: R=E+F. В© Crown copyright Met Office Andrew Lorenc 21 T пЂ1 пЂ«H R H T пЂ1 пЂЁy o пЂ H (x )пЂ© b background pdf obs likelihood function posterior analysis PDF В© Crown copyright Met Office Andrew Lorenc 22 Summary Equations - all equivalent. x minimises J пЂЁ x пЂ© пЂЅ a вЂў Variational 1 2 пЂЁx пЂ x пЂ© вЂў Kalman Filter. Kalman Gain=K. вЂў Observation space b T B пЂ1 пЂЁx пЂ x пЂ© пЂ« пЂЁy b 1 2 пЂ H пЂЁ x пЂ© пЂ© R пЂ1 пЂЁ y o пЂ H пЂЁ x пЂ© пЂ© пѓ¦ п‚¶2 J пѓ¶ пЂ1 A пЂЅ пѓ§ 2 пѓ· пЂЅ B пЂ1 пЂ« HT R пЂ1H пѓЁ п‚¶x пѓё пЂЁ x a пЂЅ xb пЂ« K y o пЂ H пЂЁ xb пЂ© K пЂЅ BH пЂЁ HBH пЂ« R пЂ© T T T o пЂ© A пЂЅ пЂЁ I пЂ KH пЂ© B пЂ1 Demonstrate equivalence using ShermanвЂ“MorrisonвЂ“Woodbury formula вЂў Model space K пЂЅ пЂЁ HT R пЂ1H пЂ« B пЂ1 пЂ© HT R пЂ1 пЂ1 B пЂЅ Z f пЂЁZ f вЂў Ensemble space Square-root Filters, e.g. ETKF T Za пЂЅ Z f T A пЂЅ Z пЂЁZ a В© Crown copyright Met Office Andrew Lorenc 23 пЂ© пЂ© a T Options in solution methods 1. Global or local вЂў Local allows observation selection, localising correlations вЂў Global avoids seams, simplifies logic, allows use of model operators & hence extends to 4D-Var. 2. Explicit or iterative вЂў вЂў Iterative is cheaper (allowing bigger problems to be solved) and can be extended to weakly nonlinear analysis. Explicit matrix inversion give analysis error covariance. Useful in QC and in some ensemble methods. В© Crown copyright Met Office Andrew Lorenc 24 Deterministic 4D-Var Initial PDF is approximated by a Gaussian. Descent algorithm only explores a small part of the PDF, on the way to a local minimum. 4D analysis is a trajectory of the full model, optionally augmented by a model error correction term. В© Crown copyright Met Office Andrew Lorenc 25 When does deterministic 4D-Var using вЂњautomaticвЂќ adjoint methods not work? Thermostats: - Fast processes which are modulated to maintain a longer-time-scale вЂњbalanceвЂќ (e.g. boundary layer fluxes). Limits to growth: - Fast processes which in a nonlinear model are limited by some available resource (e.g. evaporation of raindrops). Butterflies: - Fast processes which are not predictable over a long 4DVar time-window. (e.g. eddies with short space- & time-scales). Observations of intermittent processes: - If something (e.g. a cloud or rain) is missing from a state, then the gradient does not say what to do to make it appear. These are fundamental atmospheric processes вЂ“ it is impossible to write a good NWP model without representing them. В© Crown copyright Met Office Andrew Lorenc 26 Statistical, incremental 4D-Var PF model evolves any simplified perturbation, and hence covariance of PDF Simplified Gaussian PDF t1 Simplified Gaussian PDF t0 Full model evolves mean of PDF Statistical 4D-Var approximates entire PDF by a Gaussian. 4D analysis increment is a trajectory of the PF model, optionally augmented by a model error correction term. В© Crown copyright Met Office Andrew Lorenc 27 Statistical 4D-Var - equations Independent, Gaussian background and model errors пѓћ non-Gaussian pdf for general y: пЂЁ P пЃ¤ x,пЃ¤ О· y o пЂ© п‚µ exp пЂЁ пЂ пЂЁ пЃ¤ x пЂ пЂЁ x 1 2 пЂЁ exp пЂЁ пЂ exp пЂ Incremental linear approximations in forecasting model predictions of observed values converts this to an approximate Gaussian pdf: The mean of this approximate pdf is identical to the mode, so it can be found by minimising: Lorenc (2003a) В© Crown copyright Met Office Andrew Lorenc 28 1 2 пЂЁпЃ¤ О· пЂ« О· пЂ© 1 2 пЂЁy пЂ y пЂ© g o пЂЁ пЂx b T Q T R g пЂ©пЂ© пЂ1 пЂ1 T B пЂЁ пЂ© J пЃ¤ x,пЃ¤ О· пЂЅ пЂЁпЃ¤ x пЂ пЂЁ x пЂ« 1 2 пЂЁпЃ¤ О· пЂ« О· пЂ© пЂ« 1 2 пЂЁ g yпЂy o пЂ© T T R пЂx g пЂ©пЂ©пЂ© пЂЁ y пЂ y пЂ©пЂ© o пЂx b b g y пЂЅ H M пЃ¤ x, О· пЂ« H M x , О· 1 2 пЂЁпЃ¤ x пЂ пЂЁ x пЂЁпЃ¤ О· пЂ« О· пЂ© пЂ© пЂЁ пЂЁ пЂ© пЂ1 g пЂ©пЂ© Q пЂ1 g T пЂ1 B пЂ1 g пЂ©пЂ© пЂЁпЃ¤ x пЂ пЂЁ x пЂЁпЃ¤ О· пЂ« О· пЂ© g пЂЁy пЂ y пЂ© o b пЂx g пЂ©пЂ© Modelling and representing prior background error covariances B. вЂў Explicit point-point [multivariate] covariance functions. вЂў Transformed control variables to deal with inter-variable covariances. вЂў Vertical вЂ“ horizontal split вЂў EOF decomposition into modes. вЂў Spectral decomposition into waves. вЂў Wavelets. вЂў Recursive filters or diffusion operators to give local variations. вЂў Evolved covariances вЂ“ Ensemble members. В© Crown copyright Met Office Andrew Lorenc 29 SchlatterвЂ™s (1975) multivariate covariances Specified as multivariate 2-point functions. Not easy to ensure that specified functions are actually valid covariances. Used in OI and related observationspace methods. В© Crown copyright Met Office Andrew Lorenc 30 Transformed control variable. пѓ¦П€пѓ¶ пѓ§ пѓ· пѓ§П‡ пѓ· пЃ¤x пЂЅ U p пѓ§ пѓ· пЂЅ U p v b пѓ§ пѓ· пѓ§пЃ пѓ· пѓЁ пѓё вЂў Choose a вЂњbalancedвЂќ variable from which we can calculate balanced flow in all variables: e.g. streamfunction пЃ™. вЂў Define transforms from (U) or to (T) this variable and residual variables; by construction/hypothesis inter-variable correlations are then zero. вЂў Continue with each unbalanced residual variable in turn (velocity potential пЃЈ, pressure p, moisture пЃ) making B block diagonal. (Compare EOFs) вЂў Transformed variables still need a spatial covariance model, but not multivariate. (Further transforms are used to represent these.) B пЂЁv пЂ© пѓ¦ B пЂЁП€ пЂ© пѓ§ пѓ§ 0 пЂЅпѓ§ 0 пѓ§ пѓ§ 0 пѓЁ 0 0 B пЂЁП‡ пЂ© 0 0 B пЂЁb пЂ© 0 0 B пЂЁx пЂ© пЂЅ U p B пЂЁ v пЂ© U 0 пѓ¶ пѓ· 0 пѓ· 0 пѓ· пѓ· B пЂЁ пЃ пЂ© пѓ·пѓё T p пЃ¤ x пЂЅ U p U v U h v пЂЅ Uv. BпЂЁ v пЂ© пЂЅ I. BпЂЁпЃ¤ x пЂ© пЂЅ UUT . Lorenc, A. C., S. P. Ballard, R. S. Bell, N. B. Ingleby, P. L. F. Andrews, D. M. Barker, J. R. Bray, A. M. Clayton, T. Dalby, D. Li, T. J. Payne and F. W. Saunders. 2000: The Met. Office Global 3-Dimensional Variational Data Assimilation Scheme. Quart. J. Roy. Met. Soc., 126, 2991-3012. В© Crown copyright Met Office Andrew Lorenc 31 Estimating PDFs or covariances вЂў Even if we knew the вЂњtruthвЂќ, we could never run enough experiments in the lifetime of an NWP system to estimate its error PDF, or even its error covariance B. вЂў Simplifying assumptions are essential (e.g. Gaussian, ...) вЂў Even a simplified error model has so many parameters that we cannot determine them by NWP trials to determine which give the best forecasts. вЂў In practice we can only measure innovations вЂ“ cannot get separate estimates of B & R without assumptions (Talagrand). вЂў Need to understand physics! В© Crown copyright Met Office Andrew Lorenc 32 Effect of the null space of B вЂў B should be positive semi-definite. If it has any (near) zero eigenvalues, then no analysis increments are permitted in the direction of the corresponding eigenmodes. вЂў Causes of a null space: вЂў Strong constraints used in the definition of B. E.g. вЂў Hydrostatic & geostrophic relationships вЂў Model equation in 4D-Var вЂў Too small a sample when estimating B вЂў Effects of a null space: пЃЉ Synergistic use of complementary observations пЃЊ Spurious long-range correlations and over confidence in accuracy of analysis. В© Crown copyright Met Office Andrew Lorenc 33 Analysis error for 500hPa height for different combinations of error-free observations. Z500 1 0 0 0 h Pa h eig h t (m ) (su rface P) 1 0 0 0 -5 0 0 h Pa 5 0 0 h Pa w in d t h ickn ess (m ) co m p o n en t (layer-m ean T) (m / s) w eig h t s 0 .1 4 3 0 .4 1 9 0 .6 1 1 0 .1 9 2 0 .5 2 0 0 .8 5 3 0 .6 9 9 1 .1 4 7 0 .4 4 1 0 .6 2 8 0 .4 6 1 0 .8 8 0 В© Crown copyright Met Office Andrew Lorenc 34 V500 5 0 0 h Pa h eig h t (m ) Erro r (m ) 2 1 .0 2 0 .8 1 9 .1 1 8 .9 1 4 .4 1 8 .4 1 6 .7 1 .9 T1000-500 Z1000 Lorenc, A.C. 1981: "A global three-dimensional multivariate statistical analysis scheme." Mon. Wea. Rev., 109, 701-721. Background error (prior) covariance B modelling assumptions The first operational 3D multivariate statistical analysis method (Lorenc 1981) made the following assumptions about the B which characterizes background errors, all of which are wrong! вЂў Stationary вЂ“ time & flow invariant вЂў Balanced вЂ“ predefined multivariate relationships exist вЂў Homogeneous вЂ“ same everywhere вЂў Isotropic вЂ“ same in all directions вЂў 3D separable вЂ“ horizontal correlation independent of vertical levels or structure & vice versa. Since then many valiant attempts have been made to address them individually, but with limited success because of the errors remaining in the others. The most attractive ways of addressing them all are long-window 4D-Var or hybrid ensemble-VAR. В© Crown copyright Met Office Andrew Lorenc 36 В© Crown copyright Met Office Andrew Lorenc 36 Implications for R&D strategy of causes of improvements to NWP 3. Advanced assimilation using forecast model. вЂў 4D-Var is used by 5 of the top 7 global NWP centres. Ensemble Kalman filter methods popular in other applications. Hybrid and Ensemble-Var methods are exciting much interest in operational NWP R&D. вЂў Multiple forecasts allow the evolution of error covariances and hence the definition of a 4D covariance. вЂў The NWP model is the best tool for defining the вЂњattractorвЂќ of plausible вЂњbalancedвЂќ states. пѓћIncremental methods designed such that background is only altered based on observed evidence. В© Crown copyright Met Office Andrew Lorenc 37 Evolved covariances The Kalman Filter evolves covariances by pre- & postmultiplying by a linear forecast model. 4D-Var implicitly uses evolved covariances. В© Crown copyright Met Office Andrew Lorenc 38 P пЂЁ ti пЂ«1 пЂ© пЂЅ M i P пЂЁ ti пЂ© M iT пЂ« Qi 3D Covariances dynamically generated by 4D-Var If the time-period is long enough, the evolved 3D covariances also depend on the dynamics: B пЂЅ M n пЂ1 пЂЁ x пЂЁ tn пЂ© пЂ© T T M 1M 0 B пЂЁ xпЂЁt пЂ©пЂ© M 0 M 1 0 T M n пЂ1 Cross-section of the 4D-Var structure function (using a 24 hour window). ThГ©paut, Jean-NГ¶el, P. Courtier, G. Belaud and G LemaГ®tre: 1996 "Dynamical structure functions in a four-dimensional variational assimilation: A case study" Quart. J. Roy. Met. Soc., 122, 535-561 В© Crown copyright Met Office Andrew Lorenc 39 2-Way Coupled DA/E y n пЂЅ H (x n ), пЃі o , ... xn пЂЅ x пЂ« пЃ¤ xn f a x1 . . . a xN f f . . . п‚ пЂ UMN x OPS f N п‚ пЂ пЃ¤x N f OPS п‚ пЂ MOGREPS UM п‚ пЂ T п‚ пЂ yN F x f K п‚ пЂ y Deterministic a . . . N=23 п‚ пЂ (UM = Unified Model) (OPS = Observation Preprocessing System) x y1 E o OPS В© Crown Copyright 2011. Source: Met Office Ensemble Covariances y a a пЃ¤x1 x1 . . . п‚ пЂ a пЃ¤x N xN a пЃ¤x1 x1 UM1 a a a п‚ пЂ B 4D-VAR x a Covariances calculated directly from a sample: = = = В© Crown copyright Met Office Flavours of Ensemble Kalman Filter вЂў EnKF: closest to KF. Allows Schur product localisation. Uses perturbed observations to get correct spread. (e.g. Houtekamer & Mitchell Canada) вЂў SQRT filters: Allows Schur product localisation. Deterministic equation gives correct spread. Efficient with serial processing of obs. (e.g. Tippett, Anderson, Bishop, Hamill, Whitaker) вЂў ETKF: Localised by data selection. Deterministic equation gives correct spread. Efficient because matrices are order ensemble size. (e.g. Bowler Met Office, Kalnay, Ott, Hunt et al. Univ Maryland, Miyoshi Japan) В© Crown copyright Met Office Hybrid VAR formulation вЂў Basic code written in late 90вЂ™s! (Barker and Lorenc) вЂў VAR with climatological covariance Bc: B c пЂЅ UU пЃ¤w c пЂЅ U v пЂЅ U p U v U h v T вЂў VAR with localised ensemble covariance Pe в—‹ Cloc: пЃЎ C loc пЂЅ U U пЃЎT пЃЎ О±i пЂЅ U v О± i K 1 ( x -x ) пЃЇ О± пѓҐ K пЂ1 пЃ¤w e пЂЅ i i пЂЅ1 вЂў Note: We are now modelling Cloc rather than the full covariance Bc. вЂў Hybrid VAR: пЃ¤ w пЂЅ пЃў c пЃ¤ w c пЂ« пЃў eпЃ¤ w e J пЂЅ 1 2 В© Crown copyright Met Office Andrew Lorenc 44 v vпЂ« T 1 2 v пЃЎT пЃЎ v пЂ« Jo пЂ« Jc i Single observation tests u response to a single u observation at centre of window Horizontal Standard 3D-Var Pure ensemble 3D-Var Ensemble RMS Standard 4D-Var Adam Clayton В© Crown copyright Met Office Andrew Lorenc 45 50/50 hybrid 3D-Var The need to localise Pe вЂў Ensemble covariances are noisy. In particular, there are spurious longrange correlations: Pe вЂў Solution is to вЂњlocaliseвЂќ the covariances, by multiplying pointwise with a localising covariance Cloc: Pe в†’ Pe в—‹ Cloc: . (Lorenc 2003) Pe вЂў Crucially, localisation also increases the вЂњrankвЂќ of the ensemble covariances: the number of independent structures available to fit the observations. вЂў (No localisation implies just 23 global structures!) Errors in sampled EnKF covariances 1.5 N=100 covariance 1 0.5 0 -0.5 0 500 1000 1500 2000 2500 3000 distance (km) В© Crown copyright Met Office Andrew Lorenc 47 Page 47 Localisation: The Schur or Hadamard Product 1.5 n=100 * compact support covariance 1 0.5 0 -0.5 0 500 1000 1500 2000 2500 3000 distance (km) Page 48 The nonlinear вЂњHГіlmвЂќ humidity transform вЂў Several centres have implemented a nonlinear humidity transform to compensate for the non-Gaussian errors of humidity forecasts (HГіlm 2003, Gustafsson et al. 2011, Ingleby et al. in preparation) вЂў The вЂњprinciple of symmetryвЂќ suggests a non-Gaussian prior: пЂЁ P RH RH b пЂ© пЂЁ п‚µ exp пЂЁ RH пЂ RH пЂ© b 2 2 S пѓ©пѓ« RH , RH b пѓ№пѓ» пЂ© вЂў This makes the variational minimisation implicit; ECMWF and HIRLAM iterate this term in the outer-loop, The Met Office include it in a non-quadratic inner minimisation. В© Crown copyright Met Office Andrew Lorenc 49 Effect of 0% & 100% limits on RH This would damage forecasts of cloud and rain!! Diagram from Lorenc (2007) В© Crown copyright Met Office Andrew Lorenc 50 TRUE пѓћ вЂњbestвЂќ estimate obtained by modifying xb away from limits. --------- P(xв”‚xb) is biased, with mean given by blue line. Histogram of RH (b-o) and (b+o)/2 В© Crown copyright Met Office Andrew Lorenc 51 Principle of symmetry and HГіlm transform вЂ“ a Bayesian interpretation. What are the prior and loss function which make this optimal? вЂў The distribution of values in the background, generated by the model, is close to correct вЂ“ we have the right cloud cover on average. вЂў It is important to us to retain this correct distribution вЂ“ more so than to reduce the expected RMS error at each point. вЂў The HГіlm transform constructs a (skewed) prior whose mode is the background. вЂў We rely on a minimisation which finds this mode (not the mean) and hence returns the model background unaltered in the absence of observations. В© Crown copyright Met Office Andrew Lorenc 52 Mean b-o, classified by cloud top in b В© Crown copyright Met Office Andrew Lorenc 53 Mean b-o, classified by cloud top in o В© Crown copyright Met Office Andrew Lorenc 54 Vertical correlation with level 5 В© Crown copyright Met Office Andrew Lorenc 55 Nonlinearity вЂ“ benefitting from the attractor вЂў The atmospheric state is fundamentally governed by nonlinear effects, e.g. convective-radiative equilibrium, condensation, cloud & precipitation. Nonlinear chaotic systems have a fuzzy attractor manifold of states that occur in reality вЂ“ far fewer than all possible states. This gives us recognisable weather systems and practical weather prediction! вЂў Usual minimum variance вЂњbestвЂќ estimate is not on the attractor. вЂў The best practical way of defining the attractor in by using the full model, as we have for years in methods for spin-up and diabatic initialisation. вЂў Methods based on Gaussian PDFs can only approximate near-linear aspects of this balance. We need to add an additional prior that we want the analysis to be a state which the model might generate. вЂў It is very hard to formulate a practical Bayesian algorithm to do this. We might try engineering solutions: пѓ� Incremental methods with an outer-loop. пѓ� Multiple DA вЂњparticlesвЂќ sampling plausible solutions. В© Crown copyright Met Office Andrew Lorenc 56 В© Crown copyright Met Office Andrew Lorenc 56 Questions and answers В© Crown copyright Met Office

1/--страниц