Test all the things - Automated Testing of (RESTful) API Documentation
A presentation at APIdays Australia 2017 in in Sydney NSW, Australia by Rouven Weßling
A ut omatic t esting o f (RES T f ul) API document ation A P I
Da y s
A u s t r a li a ,
F e br u a r y
2017 B y R ouv en Weßling ( ) Ec osyst em De v eloper / De v eloper Ev ang elist , Cont entf ul @R ouv enWessling p h ot o c r ed i t :
b y
S y d ne y
H a r b ou r
B r i d g e,
A u s t r a l i a L enny
K P h ot og r a p h y (C C
B Y )
A c ont ent manag ement de v eloper platf orm with an API at its c or e.
What do y ou do ?
P o w er ed b y
A ne w look .
Document ation will be wr ong
U ser s hat e that
I t ' s
a ll i n
t h e
s pe c
api blueprint A po w erf ul high-le v el API description languag e f or w eb APIs.
Q u e s t i o n s
C o l l e c t i o n
[ / q u e s t i o n s ]
C r e a t e
a
N e w
Q u e s t i o n
[ P O S T ]
Y o u
m a y
c r e a t e
y o u r
o w n
q u e s t i o n
u s i n g
t h i s
a c t i o n .
I t
t a k e s
a
J S O N
o b j e c t
c o n t a i n i n g
a
q u e s t i o n
a n d
a
c o l l e c t i o n
o f
a n s w e r s
i n
t h e
f o r m
o f
c h o i c e s .
R e q u e s t
( a p p l i c a t i o n / j s o n )
{
" q u e s t i o n " :
" F a v o u r i t e
p r o g r a m m i n g
l a n g u a g e ? " ,
" c h o i c e s " :
[
" S w i f t " ,
" P y t h o n " ,
" O b j e c t i v e
C " ,
" R u b y "
]
}
R e s p o n s e
2 0 1
( a p p l i c a t i o n / j s o n )
H e a d e r s
L o c a t i o n :
/ q u e s t i o n s / 2
B o d y
{
" q u e s t i o n " :
" F a v o u r i t e
p r o g r a m m i n g
l a n g u a g e ? " ,
" p u b l i s h e d _ a t " :
" 2 0 1 5
0 8
0 5 T 0 8 : 4 0 : 5 1 . 6 2 0 Z " ,
" c h o i c e s " :
[
{
" c h o i c e " :
" S w i f t " ,
L e t ' s
g e t
t e s t i n g DR E DD No mor e out dat ed API document ation.
T e s t i n g
r e a d- o n ly n o d e _ m o d u l e s / . b i n / d r e d d
c m a . a p i b
h t t p s : / / a p i . c o n t e n t f u l . c o m
\
m
G E T
H o o k s b e f o r e A l l
c a l l ed
a t
t h e b e g i n ni ng
of
t h e wh ol e t es t
ru n b e f o r e E a c h
c a l l ed
b ef or e e a c h
HT T P
t r a ns a c t i on b e f o r e
c a l l ed
b ef or e s om e s p ec i f i c
HT T P
t r a ns a c t i on b e f o r e E a c h V a l i d a t i o n
c a l l ed
b ef or e e a c h
HT T P
t r a ns a c t i on i s
v a l i d a t ed b e f o r e V a l i d a t i o n
c a l l ed
b ef or e s om e s p ec i f i c
HT T P
t r a ns a c t i on i s
v a l i d a t ed a f t e r
c a l l ed
a � er s om e s p ec i f i c
HT T P
t r a ns a c t i on r e g a r d l es s
i t s
r es u l t a f t e r E a c h
c a l l ed
a � er e a c h
HT T P
t r a ns a c t i on a f t e r A l l
c a l l ed
a � er wh ol e
t es t
ru n
H o o k s n o d e _ m o d u l e s / . b i n / d r e d d
c m a . a p i b
h t t p s : / / a p i . c o n t e n t f u l . c o m
\
h o o k f i l e s
. / t e s t
h o o k s . j s
\
m
G E T
S k i ppi n g
T e s t s v a r
h o o k s
=
r e q u i r e ( ' h o o k s ' ) ;
h o o k s . b e f o r e (
" W e b h o o k
c a l l s
W e b h o o k
c a l l
d e t a i l s
G e t
t h e
w e b h o o k
c a l l
d e t a i l s " ,
f u n c t i o n
( t r a n s a c t i o n )
{
t r a n s a c t i o n . s k i p
=
t r u e ;
} ) ;
Mu t a t i n g
d a t a v a r
h o o k s
=
r e q u i r e ( ' h o o k s ' ) ;
h o o k s . b e f o r e (
" E n t r i e s
D e l e t e
a n
E n t r y " ,
f u n c t i o n
( t r a n s a c t i o n )
{
c l i e n t . c r e a t e E n t r y ( e n t r y ,
" 1 2 3 4 " )
} ) ;
R a t e
li m i t i n g v a r
h o o k s
=
r e q u i r e ( ' h o o k s ' ) ;
c o n s t
D E L A Y
=
1 0 0 0 / 6 . 0 ;
h o o k s . a f t e r E a c h ( f u n c t i o n ( t r a n s a c t i o n ,
d o n e )
{
s e t T i m e o u t ( d o n e ,
D E L A Y ) ;
} ) ;
C h a n g i n g
r e q u e s t
da t a v a r
h o o k s
=
r e q u i r e ( ' h o o k s ' ) ;
h o o k s . b e f o r e E a c h ( f u n c t i o n
( t r a n s a c t i o n ,
d o n e )
{
t r a n s a c t i o n . f u l l P a t h
=
t r a n s a c t i o n . f u l l P a t h . r e p l a c e ( ' f p 9 1 o e l s z i e a ' ,
' g b k x k l v m o l c 1 ' ) ;
d o n e ( ) ;
} ) ;
C h a i
a s s e r t i o n s v a r
h o o k s
=
r e q u i r e ( ' h o o k s ' ) ;
v a r
a s s e r t
=
r e q u i r e ( ' c h a i ' ) . a s s e r t ;
h o o k s . a f t e r E a c h ( f u n c t i o n ( t r a n s a c t i o n ,
d o n e )
{
i f
( ! t r a n s a c t i o n . s k i p )
{
a s s e r t . m a t c h ( t r a n s a c t i o n . r e a l . h e a d e r s [ ' x
c o n t e n t f u l
r e q u e s t
i d ' ] ,
/ ^ c o n t e n t
a p i : [ a
z A
Z 0
9 ] { 2 2 } $ /
}
d o n e ( ) ;
} ) ;
S e c u r i t y
�
C e n s o r
pr i v a t e
da t a v a r
h o o k s
=
r e q u i r e ( ' h o o k s ' ) ;
c o n s t
S E C R E T _ H E A D E R S
=
[ ' A u t h o r i z a t i o n ' ] . m a p ( h e a d e r
=
h e a d e r . t o L o w e r C a s e ( ) ) ;
h o o k s . a f t e r E a c h V a l i d a t i o n ( f u n c t i o n ( t r a n s a c t i o n ,
d o n e )
{
O b j e c t . k e y s ( t r a n s a c t i o n . r e q u e s t . h e a d e r s ) . f o r E a c h ( f u n c t i o n ( k e y )
{
i f
( S E C R E T _ H E A D E R S . i n d e x O f ( k e y . t o L o w e r C a s e ( ) )
1 )
{
t r a n s a c t i o n . r e q u e s t . h e a d e r s [ k e y ]
=
" * * * H I D D E N
S E C R E T
D A T A * * * " ;
}
} ) ;
d o n e ( ) ;
} ) ;
C I
t h e
be a s t
C o n c lu s i o n Base y our document ation on an API spec T est that spec Mak e it p art of y our CI Don't substit ut e y our int e gr ation t ests
Slides av ailable on Slideshar e: http://www .slideshar e.ne t /r w essling / F ollo w me on T witt er: @R ouv enWessling
Mee t up T h u r s da y ,
M a r c h
2,
2017
7: 00 pm Piv ot al Lab s, L ev el 11, 155 Clar enc e Str ee t , Sy dney https://www .mee t up . c om/Cont entf ul-U ser -Mee t up -Sy dne y / e v ents/237520492/
There are few things more frustrating than documentation that doesn't match the implementation - especially when it's documentation for your customers. In actively developed software that is a reoccurring problem. As the product evolves the documentation often lags behind and sometimes breaking changes sneak it, breaking the very examples that are part of the documentation. To avoid all these problems, it's time to start testing your docs. In this talk I'll show you what's currently possible when it comes to testing documentation, how to start testing API documentation written using API Blueprints and how to incorporate it into your workflow.
Resources
The following resources were mentioned during the presentation or are useful additional information.
