Tools to Develop

a Responsible an Ethical a Trustworthy AI

The use of systems based on Artificial Intelligence (AI) is becoming more and more common in our society. Machine Learning and Deep Learning algorithms are found from entertainment applications to support software in the health sector. However, the implementation of this emerging technology brings with it risks and social and ethical implications. Without an ethical or responsible development of the systems, this technology can undermine the autonomy, privacy or equity of people, even affecting human rights.

Taking into account the approaches for responsible development and governance of AI (ethical principles, guidelines, technical tools, among others) and the limitations when putting them into practice by more technical personnel. We present a diagram with different typologies where the stages of the AI life cycle, high-level ethical principles and the tools that can help to comply with these principles (technical and abstract resources) are distinguished, as well as other relevant information (level of development, homework, language). Our research was based on a systematic review where we found 352 resources or tools which we hope may be useful for responsible AI development, both for developers and for leaders or decision makers.

Typology

E
5
7
12
26
27
27
36
40
41
42
45
56
61
68
72
74
75
77
80
103
108
109
110
111
113
114
114
115
117
118
119
122
123
124
125
132
134
135
135
136
136
139
140
141
141
148
148
151
154
156
162
164
169
172
173
173
173
173
174
174
174
175
176
176
176
177
178
179
180
181
182
182
183
184
185
186
189
189
190
190
191
191
191
193
194
194
195
195
196
201
202
203
206
206
207
207
208
208
209
209
209
214
215
215
216
216
222
223
224
225
226
226
227
228
233
234
239
241
242
243
243
250
251
252
253
254
255
255
257
260
261
263
263
265
266
272
272
277
279
284
295
301
304
306
307
314
318
318
321
321
326
333
338
338
342
342
343
344
350
352
J
9
10
14
15
16
18
18
19
20
21
22
23
30
31
32
36
41
43
45
46
48
55
64
66
68
73
75
77
79
84
84
84
85
87
88
89
89
90
91
92
93
95
97
97
97
98
98
98
100
101
102
104
108
108
109
110
110
111
113
114
114
116
116
118
119
120
121
122
123
124
125
126
127
128
132
133
137
140
140
141
142
143
144
146
146
146
147
149
149
150
151
153
154
155
155
157
157
158
159
159
159
160
160
161
163
163
164
164
165
166
167
168
168
169
170
170
171
171
172
183
188
188
196
196
205
210
211
212
213
223
224
228
229
230
231
232
232
240
240
242
245
246
247
248
249
256
260
262
264
265
267
268
269
270
271
274
274
275
279
280
282
283
283
283
283
284
287
288
290
291
292
294
295
298
298
299
301
302
304
306
308
310
312
315
316
316
319
319
319
320
323
325
326
327
331
332
332
332
332
332
333
334
337
341
344
344
346
346
347
347
347
351
Non M
2
3
4
8
13
15
25
28
37
39
41
44
47
50
50
52
53
57
60
61
62
65
70
71
72
73
78
82
83
86
87
89
89
90
90
91
93
94
95
96
97
97
97
98
98
99
99
99
100
101
102
103
104
104
106
106
106
108
108
109
110
110
111
113
118
125
125
127
132
133
139
140
141
145
147
149
149
149
150
151
153
154
155
167
172
181
192
192
196
197
197
198
198
200
204
204
204
205
210
211
212
213
217
218
219
220
221
223
224
229
230
231
232
232
233
234
235
236
236
237
237
237
238
238
238
242
242
244
245
246
248
248
249
249
257
258
258
259
262
263
264
265
267
270
275
276
278
278
278
279
280
281
284
285
286
286
287
288
290
291
293
295
296
297
299
300
301
302
304
305
306
311
312
313
319
320
322
322
324
324
326
327
328
329
329
331
333
334
336
337
340
340
344
344
346
348
349
351
B
1
6
17
21
24
29
33
34
35
38
49
51
63
67
69
73
76
81
85
88
88
92
94
101
102
105
105
107
109
112
118
125
126
127
128
129
130
130
131
132
137
138
140
142
145
147
149
152
155
161
187
187
199
199
200
205
212
213
223
230
231
232
232
242
256
259
262
264
265
273
273
279
281
284
289
290
291
292
293
294
295
296
297
299
301
303
303
304
306
315
317
317
319
320
326
330
330
333
335
336
337
339
339
340
344
346
A
11
11
32
54
58
59
73
74
89
89
94
109
109
112
125
131
132
161
223
242
256
262
265
277
279
284
288
292
295
296
300
301
304
306
309
320
326
333
346
G
32
63
69
73
74
81
86
89
91
94
97
101
102
140
149
192
199
200
205
212
230
231
283
295
303
330

Business and problem understanding

Planning and design

Deployment and monitoring

Collection, understanding and preparation of data

Model setup and training

Performance evaluation

ID
Task
Level
Resource Title

Other title or description.

AI Life Cycle Stage

    Stage

High Level Principle

    Principles

Associated Principles

Sec. Principles

URL/DOI

Tool Type

Language

Year

Sector

ID
Task
Level
Resource Title

Other title or description.

AI Life Cycle Stage

    Stage

High Level Principle

    Principles

Associated Principles

Sec. Principles

URL/DOI

Tool Type

Language

Year

Sector

ID
Task
Level
Resource Title

Other title or description.

AI Life Cycle Stage

    Stage

High Level Principle

    Principles

Associated Principles

Sec. Principles

URL/DOI

Tool Type

Language

Year

Sector

ID
Task
Level
Resource Title

Other title or description.

AI Life Cycle Stage

    Stage

High Level Principle

    Principles

Associated Principles

Sec. Principles

URL/DOI

Tool Type

Language

Year

Sector

Task

R Regression
BC Binary Classification
MC Multi-class Classification
NLP Natural Language Processing
CV Computer Vision
C Clustering
DR Dimensionality Reduction
TS Times Series

Sector

Private
Public
NGO
Academic

Level of Development

1 Insufficient
2 Basic
3 Intermediate
4 High
5 Advanced

This section is not available on mobile devices, enter the desktop version to see the diagram and its typologies.

Cite as

If the diagram and typologies on the landing page was useful to you, you can support us by citing the article like this:

              
                @article{
                    author = {Ortega-Bolaños, Ricardo and Bernal-Salcedo, Joshua and Germán Ortiz, Mariana and Galeano Sarmiento, Julian 
                               and Ruz, Gonzalo A. and Tabares-Soto, Reinel},
                    title = {Applying the ethics of AI: a systematic review of tools for developing and assessing AI-based systems},
                    journal = {Artificial Intelligence Review},
                    volume = {57},
                    number = {110},
                    pages = {30},
                    year = {2024},
                    year={2024},
                    month={Apr},
                    day={05},
                    doi = {10.1007/s10462-024-10740-3},
                    URL = {https://doi.org/10.1007/s10462-024-10740-3}
                }