jum.2010.29.10.1423

2910jum_online.qxp:Layout 1
9/20/10
3:09 PM
Page 1423
Article
CME
Learning Curve of Emergency Physicians
Using Emergency Bedside Sonography for
Symptomatic First-Trimester Pregnancy
Timothy B. Jang, MD, Wendy Ruggeri, MD,
Pamela Dyne, MD, Amy H. Kaji, MD, PhD
Objective. The purpose of this study was to prospectively assess the learning curve of emergency
physician training in emergency bedside sonography (EBS) for first-trimester pregnancy complications.
Methods. This was a prospective study at an urban academic emergency department from August
1999 through July 2006. Patients with first-trimester vaginal bleeding or pain underwent EBS followed
by pelvic sonography (PS) by the Department of Radiology. Results of EBS were compared with those
of PS using a predesigned standardized data sheet. Results. A total of 670 patients underwent EBS for
first-trimester pregnancy complications by 1 of 25 physicians who would go on to perform at least 25
examinations. The sensitivity and specificity of EBS for an intrauterine pregnancy increased from 80%
(95% confidence interval [CI], 71%–87%) and 86% (95% CI, 76%–93%), respectively, for a physician’s first 10 examinations to 100% (95% CI, 73%–100%) and 100% (95% CI, 63%–100%) for
those performed after 40 examinations. Likewise, the sensitivity and specificity for an adnexal mass or
ectopic pregnancy changed from 43% (95% CI, 28%–64%) and 94% (95% CI, 89%–97%) to 75%
(95% CI, 22%–99%) and 89% (95% CI, 65%–98%), whereas the sensitivity and specificity for a
molar pregnancy changed from 71% (95% CI, 30%–95%) and 98% (95% CI, 94%–99%) to 100%
(95% CI, 20%–100%) and 100% (95% CI, 81%–100%). Although detection of an intrauterine or a
molar pregnancy improved with training, even with experience including 40 examinations, the sensitivity of EBS for an adnexal mass or ectopic pregnancy was less than 90%. Conclusions. There is an
appreciable learning curve among physicians learning to perform EBS for first-trimester pregnancy
complications that persists past 40 examinations. Key words: ectopic pregnancy; first-trimester pregnancy; intrauterine pregnancy; learning curve.
Abbreviations
ACEP, American College of Emergency Physicians; CI,
confidence interval; EBS, emergency bedside sonography; ED, emergency department; IUP, intrauterine pregnancy; OR, odds ratio; PGY, postgraduate year; PS,
pelvic sonography; SAEM, Society for Academic
Emergency Medicine
Received April 20, 2010, from the Department of
Emergency Medicine, David Geffen School of
Medicine, Olive View Medical Center and UCLA
Medical Center, Sylmar, California USA (T.B.J., W.R.,
P.D.); and Department of Emergency Medicine,
David Geffen School of Medicine, Harbor-UCLA
Medical Center, Torrance, California USA (T.B.J.,
A.H.K.). Revision requested May 17, 2010. Revised
manuscript accepted for publication May 26, 2010.
Address correspondence to Timothy B. Jang, MD,
Department of Emergency Medicine, David Geffen
School of Medicine, Olive View Medical Center and
UCLA Medical Center, 14445 Olive View Dr, North
Annex, Sylmar, CA 91342 USA.
E-mail: tbj@ucla.edu
CME
Article includes CME test
T
he training of clinicians performing bedside
sonography in the emergency department (ED)
has been controversial, and there are different
guidelines for their instruction.1–5 Although
several studies have described the learning curve of clinicians performing bedside sonography for the evaluation of biliary disease and trauma,4–8 we are unaware of
any study assessing the learning curve of physicians
training in emergency bedside sonography (EBS) for the
evaluation of first-trimester pregnancy complications.
Therefore, the purpose of this study was to prospectively
assess the learning curve of physicians training in EBS for
first-trimester pregnancy complications.
© 2010 by the American Institute of Ultrasound in Medicine • J Ultrasound Med 2010; 29:1423–1428 • 0278-4297/10/$3.50
2910jum_online.qxp:Layout 1
9/20/10
3:09 PM
Page 1424
Learning Curve for Emergency First-Trimester Obstetric Sonography
Materials and Methods
Study Design
This was an Institutional Review Board–
approved prospective cohort study of emergency
physicians training in EBS, using a convenience
sample of patients enrolled between August 1,
1999, and July 31, 2006. Patients consented and
underwent EBS before pelvic sonography (PS)
by the Department of Radiology for the evaluation of first-trimester pregnancy complications.
The study physicians performed EBS to detect
sonographic signs of intrauterine and ectopic
pregnancies, and results were recorded on a predesigned data sheet.
Study Setting
This study was conducted at an urban academic
ED with 49,000 annual adult visits, a postgraduate year 2 (PGY-2) through PGY-4 emergency
medicine residency and a PGY-1 through PGY5 combined internal medicine/emergency
medicine residency.
Selection of Participants
The participating physicians were emergency
physicians training in EBS and had completed a
2-day introductory course on emergency sonography, which included 2 EBS examinations on
healthy pregnant volunteers. Both attending and
resident physicians were included because the
emergency ultrasound program was started in
1999, and none of the physicians at our institution met the training guidelines of the American
College of Emergency Physicians (ACEP) or the
Society for Academic Emergency Medicine
(SAEM) regarding obstetric sonography.
All patients presenting to the ED with firsttrimester vaginal bleeding, pain, or nausea and
vomiting were eligible for participation if their
treating physicians were ordering PS for firsttrimester pregnancy complications. Patients
were excluded if they could not speak English,
had known results of PS done within 60 days, or
were otherwise unable to give informed consent.
Protocol
Participating physicians obtained consent from
the patients and performed EBS to detect specific sonographic signs of intrauterine and ectopic
1424
pregnancies and recorded their results before PS
was performed. Research assistants, trained in
data abstraction and blinded to the results of the
EBS examinations, then reviewed the results of
PS read by board-certified radiologists from the
Division of Ultrasound and Mammography
within the Department of Radiology for subsequent comparison.
Emergency bedside sonographic examinations
done with the assistance of a more experienced
physician were tracked by the examination experience of the senior physician, but EBS examinations done for training purposes after PS was
performed in the Department of Radiology were
excluded. These training examinations were
tracked only for the purpose of assessing the
overall experience of the physician. All EBS
examinations were reviewed for education and
feedback within 72 hours by a sonographer certified by the American Registry for Diagnostic
Medical Sonography from 1999 to 2001 and a
physician who met the ACEP and SAEM training
guidelines from 2001 to 2006. The results of these
reviews were not used to alter the completed
data sheets because the performance of EBS
depends on the technical and interpretive skills
of the operator.
Study Measurements
Emergency bedside sonographic examinations
were performed using an SSD-1400 system
(Aloka Co, Ltd, Tokyo, Japan) with a 3- to 5-MHz
curvilinear probe and a 7.5-MHz endocavitary
probe and involved both transabdominal and
endovaginal imaging to evaluate the following:
(1) the presence of an intrauterine pregnancy
(IUP), defined as a gestational sac with a yolk sac
or the presence of an intrauterine fetal pole; (2)
the presence of a molar pregnancy, defined sonographically by a “cluster of grapes” or “snowstorm” appearance; and (3) the presence of a
definite ectopic pregnancy, defined as an
extrauterine fetal pole or gestational sac with a
yolk sac, or an abnormal adnexal mass, defined
as an abnormal-appearing echogenic mass in
the adnexa, an “adnexal ring” (ie, trophoblastic
reaction in the fallopian tube), or an “interstitial
line sign” (ie, cavitary line formed by myometrium pointing to an ectopic focus) of concern for a
possible ectopic pregnancy. A simple ovarian cyst
J Ultrasound Med 2010; 29:1423–1428
2910jum_online.qxp:Layout 1
9/20/10
3:09 PM
Page 1425
Jang et al
was not considered an abnormal adnexal mass
(see Appendix). The criterion standard for each
of these measures was the final reading of PS by
board-certified radiologists blinded to the EBS
results, which was performed using an iU22 system (Philips Healthcare, Bothell, WA) with an
endocavitary probe.
Data Analysis
Data were collected in an Excel database
(Microsoft Corporation, Redmond, WA) and
translated into a native SAS format using DBMS/
Copy (Dataflux Corporation, Cary, NC). Analyses
were conducted using SAS version 9.1 (SAS
Institute Inc, Cary, NC). The sensitivity and
specificity of EUS were calculated using 95%
confidence intervals (CIs) to assess both statistical significance and the clinical effect.9
It was predetermined to track the EBS examinations per physician by experience level in
blocks of 10 (ie, 1–10, 11–20, 21–30, 31–40, and
>40 examinations). Thus, the data were hierarchical such that the second block of 10 EBS
examinations for every physician was in group 2
(11–20); the third block for every physician was
in group 3 (21–30); the fourth block for every
physician was in group 4 (31–40); and the fifth
block for every physician was in group 5 (>40).
Consequently, the number of physicians in each
successive group was anticipated to be smaller
than that in the preceding group. Furthermore, it
was predetermined to include only the data from
the physicians who completed at least 25 training examinations, consistent with the ACEP minimum training guideline.10
Mantel-Haenszel χ2 testing was done to assess
for changes in performance between each
group. Finally, the GENMOD procedure was
used to perform a clustered regression analysis
to account for the fact that examinations by
one operator are typically more similar than
those performed by another operator; thus, the
data analysis took intraoperator cluster correlation into account rather than assuming independence among all observations. Adequacy of
the regression model fit was assessed by using
the generalized score statistic criterion (P > .05)
and quasi-likelihood information criterion,
which is a modification of the Akaike information criterion.
J Ultrasound Med 2010; 29:1423–1428
Results
A total of 670 patients underwent EBS for firsttrimester pregnancy complications by 1 of 25
physicians who performed more than 25 examinations. The sensitivity and specificity of EBS for
an IUP, a molar pregnancy, and an adnexal mass
or ectopic pregnancy are shown in Table 1, showing an overall improvement with training.
However, even with experience including 40
examinations, the sensitivity of EBS for an adnexal mass or ectopic pregnancy was less than 90%.
The Mantel-Haenszel χ2 test for trend showed a
statistically significant increase for true-positive
identification of an IUP (χ2 = 8.6; P = .0034), as
well as a statistically significant decrease in falsepositive identification of an IUP (χ2 = 4.1; P = .04),
false-negative identification of an IUP (χ2 = 9.3;
P = .002), and false-negative identification of an
adnexal mass (χ2 = 9.1; P = .003). However, there
was no statistically significant increase in trend
for true-negative identification of an IUP (χ2 =
0.3; P = .6), true-positive identification of an
adnexal mass (χ2 = 2.6; P = .1), or true-positive
(χ2 = 1.1; P = .3) or true-negative (χ2 = 0.2; P = .6)
identification of a molar pregnancy, nor was
there a statistically significant decreasing trend
Table 1. Sensitivity and Specificity of EBS for Specific Findings
Condition, n (%)
IUP (n = 408 total)
108 (57)
125 (62)
104 (59)
57 (70)
14 (61)
Adnexal mass (n = 103 total)
31 (16)
30 (15)
24 (14)
14 (17)
4 (17)
Molar pregnancy (n = 29 total)
7 (4)
10 (5)
7 (4)
3 (4)
2 (9)
Group (n)
Sensitivity, %
(95% CI)
Specificity, %
(95% CI)
1 (188)
2 (202)
3 (175)
4 (82)
5 (23)
80 (71–87)
83 (75–89)
92 (85–96)
93 (82–98)
100 (73–100)
86 (76–93)
91 (82–96)
94 (85–98)
92 (72–99)
100 (63–100)
1 (188)
2 (202)
3 (175)
4 (82)
5 (23)
43 (28–64)
63 (44–79)
100 (78–100)
79 (49–94)
75 (22–99)
94 (89–97)
96 (91–98)
100 (97–100)
100 (93–100)
89 (65–98)
1 (188)
2 (202)
3 (175)
4 (82)
5 (23)
71 (30–95)
80 (44–96)
100 (56–100)
100 (31–100)
100 (20–100)
98 (94–99)
98 (95–99)
100 (96–100)
100 (94–100)
100 (81–100)
A total of 670 patients were enrolled: 408 with IUPs, 103 with adnexal
masses or ectopic pregnancies, 29 with molar pregnancies, and 130
without definite IUPs, adnexal masses or ectopic pregnancies, or molar
pregnancies.
1425
2910jum_online.qxp:Layout 1
9/20/10
3:09 PM
Page 1426
Learning Curve for Emergency First-Trimester Obstetric Sonography
for false-negative identification of an adnexal
mass (χ2 = 3.7; P = .06) or false-positive (χ2 = 3.2;
P = .07), or false-negative (χ2 = 2.2; P = .1) identification of a molar pregnancy.
When using a random effects model and cluster
regression analysis to account for intraoperator
correlation, for every incremental increase in 10
examinations, there was a statistically significant increase in true-positive identification of
an IUP (odds ratio [OR], 1.13; 95% CI, 1.06–1.22;
P = .0006) and an adnexal mass/ectopic pregnancy (OR, 1.10; 95% CI, 1.03–1.17; P = .005).
There was also a statistically significant
decrease in false-negative identification of an
IUP (OR, 0.79; 95% CI, 0.68–0.93; P = .004) and
an adnexal mass/ectopic pregnancy (OR= 0.78;
95% CI, 0.64–0.96; P = .02), as well as false-positive
(OR, 0.75; 95% CI, 0.56–0.96; P = .02) and falsenegative (OR, 0.73; 95% CI, 0.55–0.96; P = .02)
identification of a molar pregnancy. However,
there was no significant increase in true-negative identification of an IUP (OR, 0.97; 95% CI,
0.93–1.02; P = .2), true-negative identification of
an adnexal mass/ectopic pregnancy (OR, 1.02;
95% CI, 0.94–1.12; P = .5), or true- positive (OR,
1.1; 95% CI, 0.99–1.22; P = .07) or true-negative
(OR, 0.98; 95% CI, 0.11–1.1; P = .7) identification
of a molar pregnancy, nor was there a decrease
in false-positive identification of an IUP (OR,
0.86; 95% CI, 0.71–1.05; P = .1) or false-positive
identification of an adnexal mass/ectopic pregnancy (OR, 0.87; 95% CI, 0.69–1.09; P = .2).
Discussion
Several different guidelines have been proposed
for training clinicians to perform various sonographic examinations outside the radiology
suite.1–3,5 The American Institute of Ultrasound
in Medicine recommends at least 300 sonographic examinations for obstetric indications.11
However, because EBS is focused and goal oriented, being done at the point of care, rather
than comprehensive and extensive, being done
in the radiology suite, the ACEP recommends a
minimum of 25 examinations for training in each
EBS indication,10 consistent with similar findings
from the family medicine12 and obstetric nursing13 literature.
1426
Our data suggest that the detection of IUPs with
EBS among women presenting to the ED with
first-trimester complications improves with
training but requires more than the 15 examinations suggested by data regarding focused
sonography done by obstetric nurses in later
pregnancy13. We suspect that the need for
increased training is due to the difficulty of making an accurate diagnosis in early pregnancy as
opposed to visualizing a large intrauterine fetus
in later pregnancy. In our study sample, the ability to detect IUPs at earlier gestational ages
improved with training. In group 1, there were 22
false-negative results for an IUP, with a mean gestational age of 5 weeks 1 day, compared with
groups 4 and 5, where there were 4 false-negative
results for an IUP, with a mean gestational age of
4 weeks 1 day. In groups 4 and 5, there were 19
pregnancies detected with gestational ages
between 3 and 6 weeks.
On the other hand, although detection of an
IUP improved with training, even with experience including 40 examinations, the sensitivity of
EBS for an abnormal adnexal mass or ectopic
pregnancy was less than 90%. This suggests that
even more training in EBS is required to rule out
an ectopic pregnancy.
Several studies in the literature have shown that
emergency physicians could perform EBS for the
diagnosis of an IUP and expedite care in the
ED.14–16 Our data are consistent with these prior
studies but clearly show a learning curve for
acquiring such skills. Although our data suggest an
asymptotic leveling of the learning curve at around
40 examinations, the low number of physicians
with more than 25 examinations limits the conclusions that can be drawn. In our experience,
physicians gain facility with EBS in the 26- to 40examination range that allows them to reliably
detect an IUP. However, because the diagnosis of
an ectopic pregnancy can be more difficult due to
subtleties related to imaging of the adnexa, training
including 50 to 75 examinations is required before
the sensitivity approaches 90% for an ectopic pregnancy. Therefore, our institutions require more than
25 EBS examinations, a minimum of 4 IUPs at less
than 8 weeks’ gestation, a minimum of 4 ectopic
pregnancies, and a minimum accuracy rate of 90%
for privileges to perform EBS for first-trimester
pregnancy complications.
J Ultrasound Med 2010; 29:1423–1428
2910jum_online.qxp:Layout 1
9/20/10
3:09 PM
Page 1427
Jang et al
Because the Residency Review Committee for
emergency medicine only required 40 total
sonographic examinations per resident before
2007, it is not surprising that few physicians in
our sample performed more than 25 EBS
examinations. This was also the case in recent
studies of emergency sonography for biliary
disease and trauma.5,8,17 This could be
improved with a multicenter study examining
the changes in competency that occur during an
emergency ultrasound fellowship, in which fellows are expected to perform 1000 sonographic
examinations and more than 150 obstetric
examinations.
This study had several limitations. First, only 25
of 137 eligible physicians (18%) performed more
than 25 examinations, giving us a small sample
of physicians who met the ACEP training guidelines. This represents an “ultrasound interest”
bias on the part of the physicians who were
clearly interested in performing the examinations and knew that their results were being
studied, raising the potential for a Hawthorne
effect. Our findings may not apply to other
physicians because sonography is operator
dependent, and poor skills may persist despite
training including up to 200 examinations.18 We
suspect this is one reason why the Residency
Review Committee for emergency medicine
recently changed the EBS training requirement
from a minimum of 40 total examinations to
documentation of competency.19
Second, whereas hands-on experience and the
number of examinations are related to competency, they may not be the most important factors in developing competency. It may be that
the number of examinations with positive findings is more important than the total number of
examinations performed. That was not assessed
in this study but should be addressed in the
future.
Third, this was a single-center study with a particular training and quality assurance protocol.
Emergency medicine residencies with different
resources (eg, number of ultrasound machines
and available CD learning modules), personnel
(eg, registered diagnostic medical sonographers
and fellowship-trained ultrasound directors),
and training requirements (eg, hours of didactic
training, amount of hands-on teaching, and
J Ultrasound Med 2010; 29:1423–1428
required number of sonographic examinations
for graduation) might have different learning
curves. This could be improved with a multicenter study in the future. Unfortunately, whereas
the ACEP and SAEM have clear guidelines, fewer
than one-third of residencies meet them,20
which would make it difficult to control for such
variations in training.
Finally, sonography is operator dependent and
depends on the “inherent ability of trainees to
learn sonographic skills.”18 That was not the
focus of this investigation but should be studied
further. In a study of radiology residents, every
resident had a learning curve, but those who
started out with low competency scores continued to have poor performance, even with training and experience including up to 200
examinations.18 However, when addressing minimum training standards, requirements should
be set at a level to ensure that most physicians,
not just the naturally adept, can competently
perform EBS. Our data suggest that a minimum
standard of 25 examinations is inadequate for
ensuring competency in EBS for the diagnosis of
ectopic pregnancy.
In conclusion, there is an appreciable learning
curve among physicians learning to perform EBS
for first-trimester pregnancy complications that
persists past 40 examinations.
Appendix: Data Sheet Items
Identifying information:
Patient name______________________________________________
Hospital number __________________________________________
Date of service ____________________________________________
Physician-sonographer name ________________________________
Sonographic findings:
___ IUP (EGA ___ weeks ___ days by ___ GS with yolk sac __ CRL __
BPD __ FL)
___ Ectopic pregnancy (extrauterine gestational sac with yolk sac or
fetal pole) or __ abnormal adnexal mass
___ Molar pregnancy
___ None of the above
BPD indicates biparietal diameter; CRL, crown-rump length; EGA, estimated gestational age; FL, femur length; and GS, gestational sac.
1427
2910jum_online.qxp:Layout 1
9/20/10
3:09 PM
Page 1428
Learning Curve for Emergency First-Trimester Obstetric Sonography
References
1.
1428
Robinson NA, Clancy MJ. Should UK emergency physicians
undertake diagnostic ultrasound examinations? J Accid
Emerg Med 1999; 16:248–249.
2.
Bullock C. AIUM and SAEM square off over ultrasound.
Emerg Med News 1993; 15:1.
3.
Lanoix R. Credentialing issues in emergency ultrasonography. Emerg Med Clin North Am 1997; 15:913–920.
4.
Shackford SR, Rogers FB, Osler TM, Trabusky ME, Clauss
DW, Vane DW. Focused abdominal sonogram for trauma:
the learning curve of nonradiologist clinicians in detecting
hemoperitoneum. J Trauma 1999; 46:553–564.
5.
Jang T, Sineff S, Naunheim R, Aubin C. Residents should
not independently perform focused abdominal sonography for trauma after 10 training examinations. J Ultrasound
Med 2004; 23:793–797.
6.
Smith RS, Kern SJ, Fry WR, Helmer SD. Institutional learning curve of surgeon-performed trauma ultrasound. Arch
Surg 1998; 133:530–536.
7
Ma OJ, Mateer JR, Ogata M, Kefer MP, Wittmann D,
Aprahamian C. Prospective analysis of a rapid trauma ultrasound examination performed by emergency physicians.
J Trauma 1995; 38:879–885.
8.
Jang T, Aubin C, Naunheim R. Minimum training for right
upper quadrant ultrasonography. Am J Emerg Med 2004;
22:439–443.
9.
Hanley JA. The place of statistical methods in radiology
(and in the bigger picture). Invest Radiol 1989; 24:10–16.
10.
American College of Emergency Physicians Board of
Directors. ACEP emergency ultrasound guidelines, 2008.
American College of Emergency Physicians website.
http://www.acep.org/WorkArea/DownloadAsset.aspx?id=
32878. Accessed August 31, 2009.
11.
American Institute of Ultrasound in Medicine. Training
guidelines for physicians who evaluate and interpret diagnostic ultrasound examinations. American Institute of
Ultrasound in Medicine website. http://www.aium.org/
publications/statements.aspx. Accessed August 31, 2009.
12.
Dresang LT, Rodney WM, Dees J. Teaching prenatal ultrasound to family medicine residents. Fam Med 2004;
36:98–107.
13.
Stringer M, Miesnik SR, Brown LP, Menei L, Macones GA.
Limited obstetric ultrasound examinations: competency
and cost. J Obstet Gynecol Neonatal Nurs 2003; 32:
307–312.
14.
Shih CHY. Effect of emergency physician-performed pelvic
sonography on length of stay in the emergency department. Ann Emerg Med 1997; 29:348–352.
15.
Burgher SW, Tandy TK, Dawdy MR. Transvaginal ultrasonography by emergency physicians decreases patient
time in the emergency department. Acad Emerg Med
1998; 5:802–807.
16.
Blaivas M, Sierzenski P, Plecque D, Lambert M. Do emergency physicians save time when locating a live intrauterine pregnancy with bedside ultrasonography? Acad Emerg
Med 2000; 7:988–993.
17.
Ma JO, Gaddis GM, Robinson LH, et al. Accuracy of FAST
examination interpretation as influenced by anechoic stripe
size and physician experience level [abstract]. Acad Emerg
Med 2004; 11(suppl):581.
18.
Hertzberg BS, Kliewer MA, Bowie JD, et al. Physician training requirements in sonography: how many cases are
needed for competence? AJR Am J Roentgenol 2000;
174:1221–1227.
19.
Accreditation Council for Graduate Medical Education.
Emergency medicine guidelines. Accreditation Council for
Graduate Medical Education website. http://www.
acgme.org/acWebsite/RRC_110/110_guidelines.asp#res.
Accessed August 31, 2009.
20.
Bakhtiari P, Gough JE, Brewer KL. Do emergency medicine
residency programs meet American College of Emergency
Physicians requirement of ultrasound curricula [abstract]?
Ann Emerg Med 2005; 46(suppl):S79.
J Ultrasound Med 2010; 29:1423–1428