A MORRER DA CURA

 

No consultório Canhoto, Rui Pena Pires, o sociólogo que dirige um pink-tank, esclarece diariamente dúvidas sobre a governação socialista.

Há dias, apareceu-lhe um colega cismado com a mercantilização do ensino superior. Diagnosticou-lhe uma forma qualquer de umbiguismo e mandou-o consultar a bula, na esperança de que engula de vez o remédio para a universidade. Isso, ou passar a olhar para outro umbigo.

Intrigante é que, sobre o remédio prescrito, garanta que este jamais convertirá alunos em clientes, quando já estão por aí documentados efeitos indesejados.

 

Ainda os AERES que vêm de França

AERES : les chercheurs malpubliants vus par des malvoyants

Vos papiers! La science à l’aune de la raison comptable

Journals under Threat : A Joint Response from HSTM Editors

“The State We’re In” : le classement des revues par l’AERES

On a perdu la 71ème section !!

UBU s’invite au NRS pour évaluer les chercheurs

Y a-t-il un Nature ou un Science en SHS?

L’évaluation des enseignants chercheurs, critiques et propositions

La fièvre de l’évaluation de la recherche. Du mauvais usage des faux indicateurs

Já suspeitávamos (parem as rotativas!)

Adrian Johnson

PUBLISH AND BE WRONG

One group of researchers thinks headline-grabbing scientific reports are the most likely to turn out to be wrong

In economic theory the winner’s curse refers to the idea that someone who places the winning bid in an auction may have paid too much. Consider, for example, bids to develop an oil field. Most of the offers are likely to cluster around the true value of the resource, so the highest bidder probably paid too much.

The same thing may be happening in scientific publishing, according to a new analysis. With so many scientific papers chasing so few pages in the most prestigious journals, the winners could be the ones most likely to oversell themselves—to trumpet dramatic or important results that later turn out to be false. This would produce a distorted picture of scientific knowledge, with less dramatic (but more accurate) results either relegated to obscure journals or left unpublished.

In Public Library of Science (PloS) Medicine, an online journal, John Ioannidis, an epidemiologist at Ioannina School of Medicine, Greece, and his colleagues, suggest that a variety of economic conditions, such as oligopolies, artificial scarcities and the winner’s curse, may have analogies in scientific publishing.

Dr Ioannidis made a splash three years ago by arguing, quite convincingly, that most published scientific research is wrong. Now, along with Neal Young of the National Institutes of Health in Maryland and Omar Al-Ubaydli, an economist at George Mason University in Fairfax, Virginia, he suggests why.

It starts with the nuts and bolts of scientific publishing. Hundreds of thousands of scientific researchers are hired, promoted and funded according not only to how much work they produce, but also to where it gets published. For many, the ultimate accolade is to appear in a journal like Nature or Science. Such publications boast that they are very selective, turning down the vast majority of papers that are submitted to them.

Picking winners

The assumption is that, as a result, such journals publish only the best scientific work. But Dr Ioannidis and his colleagues argue that the reputations of the journals are pumped up by an artificial scarcity of the kind that keeps diamonds expensive. And such a scarcity, they suggest, can make it more likely that the leading journals will publish dramatic, but what may ultimately turn out to be incorrect, research.

Dr Ioannidis based his earlier argument about incorrect research partly on a study of 49 papers in leading journals that had been cited by more than 1,000 other scientists. They were, in other words, well-regarded research. But he found that, within only a few years, almost a third of the papers had been refuted by other studies. For the idea of the winner’s curse to hold, papers published in less-well-known journals should be more reliable; but that has not yet been established.

The group’s more general argument is that scientific research is so difficult—the sample sizes must be big and the analysis rigorous—that most research may end up being wrong. And the “hotter” the field, the greater the competition is and the more likely it is that published research in top journals could be wrong.

There also seems to be a bias towards publishing positive results. For instance, a study earlier this year found that among the studies submitted to America’s Food and Drug Administration about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results were. But negative results are potentially just as informative as positive results, if not as exciting.

The researchers are not suggesting fraud, just that the way scientific publishing works makes it more likely that incorrect findings end up in print. They suggest that, as the marginal cost of publishing a lot more material is minimal on the internet, all research that meets a certain quality threshold should be published online. Preference might even be given to studies that show negative results or those with the highest quality of study methods and interpretation, regardless of the results.

It seems likely that the danger of a winner’s curse does exist in scientific publishing. Yet it may also be that editors and referees are aware of this risk, and succeed in counteracting it. Even if they do not, with a world awash in new science the prestigious journals provide an informed filter. The question for Dr Ioannidis is that now his latest work has been accepted by a journal, is that reason to doubt it?

Economist, 9 Out. 2008

Ler os comentários ao artigo AQUI

ÚLTIMA LIÇÃO

I’m Leaving

By John Smith

Inside Higher Ed, 31 de Outubro de 2008

The withdrawal of the last Russian regiment in Afghanistan

I distinctly recall the first day of graduate school. Some of my classmates knew the field’s top-tier journals, the term “anonymous peer reviewing,” and each professor’s research area of expertise. I was a neophyte with raw, analytical skills, no publications, a healthy ego and a desire to teach at a small, liberal arts college, much like my alma mater. I soon learned my discipline – the jargon, the journals and the gossip.

I honed my writing skills, and, more important, my thinking skills. Yet for all the merits of graduate school, even the premier one from which I was graduated, I left disappointed and ambivalent about the process. I took some classes with engaged, brilliant and dedicated professors, but I also attended more than a few seminars with detached scholars who thought of students as distractions from their labs and research. They were famous, but they could not teach, even their own research.

Like many other graduate students, I slogged through the bad, and made the most of the good. I got the job at the liberal arts college, where I received tenure, and even served as a department chair (a burden, not an honor, I tell you). I now want out.

Why? Because I fear that I have become the archetypical professor whom I did not want to become.

Don’t get me wrong. I still prepare my lectures and judging from the teacher evaluations, I know that I make students better thinkers. The classroom give-and-take produces a high that cannot be easily described or imitated. Even more, I love doing research. Sitting with pen and book in hand, or typing after months of textual analysis, is a rare joy. You mean I get paid to think? About ideas that inspire me? And I can read other research it, and dissect its merits? This gig is too good to be true.

Bingo.

After too many years at this job (I am in my mid-40s), I have grown to question higher education in ways that cannot be rectified by a new syllabus, or a sabbatical, or, heaven forbid, a conference roundtable. No, my troubles with this treasured profession are both broad and deep, and they begin with a fervent belief that most of today’s college students, especially those that come to college straight from high school, are unnecessarily coddled. Professors and administrators seek to “nurture” and “engage” and they are doing so at the expense of teaching. The result: a discernable and precipitous decline in the quality of college students. More of them come to campus with dreadful study habits. Too few of them read for pleasure. Too many drink and smoke excessively. They are terribly ill-prepared for four years of hard work, and most dangerously, they do not think that college should be arduous. Instead they perceive college as an overnight recreation center in which they exercise, eat, and in between playing extracurricular sports, they carry books around. If a professor is lucky, the books are being skimmed hours before class.

How do I know that my concerns are not unique to my employer, or my classroom? My students are brutally honest – they tell me with candor and without shame that their peers think of college as a four year cruise without a destination.

No doubt these students deserve some blame for their lethargy, but some culpability lies with their professors, and the administrators who ostensibly but unsuccessfully provide vision and direction. Today’s faculty and administrators capitulate to students’ demands in innumerable ways. They hold classes outside on sunny days, not really caring if there is no blackboard, or if the students are admiring each other instead of the texts to be dissected. They encourage students to think of college as a “comfortable” and “supportive” community, not as a means to acquire necessary skills. Far too many of my colleagues are dialing in – showing up late, popping in videos during class, assigning group projects, or sitting in a circle and asking students how they feel. Why they have abandoned classroom rigor is something that only they can answer. But one answer is simple – students flock to these popular classes, probably because they cater to the students’ worst sensibilities. Homework is minimal, or sometimes optional. Surprise quizzes are considered unfair. Late assignments are not failed. Some grades are even negotiable.

Such a pedagogy runs counter to the school, undergraduate and graduate training that I received, but it is openly embraced by nervous administrators who encourage faculty members to be innovative, experimental and experiential. They speak openly about pandering to student demands, but opt not to use the word “pander,” employing instead the curious and the trendy phrase “student empowerment.” I prefer to empower them with reading skills. But such a mission is considered old-fashioned. Maybe I should attend a seminar (don’t worry, the college will pay for it) titled “Technology in the classroom” or “Innovative pedagogies in the 21st century.” I pass.

Grade inflation is rampant. Students think of a “B minus” as an F. I constantly get criticized for grading too harshly, even though I find my mean grade point average has risen over the past decade. A “C” to today’s student is unfathomable. “Professor, I am on scholarship. How can you give me a C?” I remind them that I do not “give’” grades, but such semantics are lost on the student who yearns for an A at any cost. I tell them that I got Bs and Cs and I never complained, because I knew I deserved them. They do not believe me. (Maybe I should post my undergraduate and graduate transcripts on my office door?)

Grades did not matter to me because I believed in the superiority of my professor’s judgment. I recall questioning a professor’s grade – once and only once, only after I showed the assignment and his comments to a senior who lived down the hall. She advised me to speak to the professor. I did. The professor had made a simple calculating mistake, and apologized for his error. We remain friends to this day.

Today’s students are not questioning the logic behind the grades; They are questioning why their grades in my class are lower than in their other classes. Down the hall, those same students can get an A- by putting in three hours of work a week. How do I know? The students tell me, candidly, and without shame or the slightest pangs of guilt. To them, this disparity just doesn’t seem fair, and is the fault of the tougher grader.

Higher education for too many undergraduates at too many liberal arts colleges has become a puffy sofa nestled with down pillows. For a few bucks and in a few hours, students can take a test and learn that they are language disabled, or mathematically disabled, or for a few bucks more, both. Students increasingly ask me during advising sessions if a class is tough or hard, or if the professor assigns a lot of reading, because they need to “lighten their load.” “I want to take a class with Professor So-And-So. I have a lot on my mind, and I don’t want to stress out.” “Don’t worry,” I say, “you won’t.”

This comfy zone of mediocrity extends beyond the classroom. “Student life” largely serves to debilitate the notion of a genuine, deliberative, academic community. Rather than fuel cerebral discussions with activities for the mind, resident advisors and their adult supervisors plan activities that redefine anti-intellectualism. There is Sensitivity Day, Tolerance Day, and Wear [insert color here] Day, and a host of other events that are aimed at “inspiring.” Dorm life is supposed to be cool, fun and engaging. For me, it was simply a place to sleep.

My faculty colleagues rarely complain about their daily lives, or about the state of higher education. To the contrary, they feed the mindset that all students are exceptional by awarding high grades, honors and special prizes to the intellectually inferior. These faculty also yearn to be comfortable. How? By immersing themselves in trivial pursuits, like how many members should serve in the faculty senate, or whether serving on the Education Policy Committee should be determined by a simple majority, or a run-off election.

Intellectual sparring (dare I use the term) about ideas – among students and faculty – has been replaced by one-sided, partisan drivel (for example, Obama = admirable. McCain = terrible and, for the record, I will be voting for Obama). While it would be easy to blame a liberal bias among faculty for this groupthink, it should be noted that this simple world of good and bad pervades the world around us. On radio, television and the Internet, ideological pundits scream at one another with vitriol and fervor. My partisan colleagues are universally National Public Radio listeners. They do not hear the other side, so it is easy to demonize the other side. Their students are listening, and sadly think of conservatism in its many forms as horrific. Worse still, they now conflate liberal passion and advocacy with justice, and by default, analytic rigor and reason. They do not weigh evidence, or take note of pro, cons, costs or benefits. Doing so would be to admit that there are merits to positions they do not hold. To acknowledge that their ideology is imperfect is the first step towards compromise, or in their overused, precious phrase, “selling out.”

Their idealism, of course, is a work in progress. Nonprofit employment is admirable, but doing the same work for a for-profit corporation (with health care and retirement benefits) is deemed suspicious. Yet when college is completed, too many graduates have trouble finding work. The economy is rough, and even rougher for math-disabled, language-disabled, ideologically-driven, emotive students who do not read for pleasure. Should they take, say, an accounting course, or Shakespeare, either of which would test or push their comfort zones? Their hearts say yes, but the problem is that these classes meet early in the morning (Shakespeare at 8:30 am? C’mon!), when hangovers are to be nursed and sleepy minds are not to be awakened. Besides, rumor is that the Shakespeare professor is a tough grader.

Working at a small college is no easy task. We professors oftentimes work without research assistants. We have heavy teaching loads, and we grade our own assignments. Endowments are low, and so too are salaries and research funding. But hard work need not be depressing, and rather than become depressed, I have learned after almost 20 years that I am woefully ill-suited for today’s classroom.

Will I miss some of my colleagues? Sure. They have a remarkable ability to enjoy their craft, but I have great difficulty believing that I am making a significant difference in the lives of my students. Are my peers aware students are skimming the reading? Yes. They have figured out that getting emotionally invested in the student body is both taxing and fruitless. Instead they enjoy their autonomy and the bucolic campus life without a second thought, or with a deeply imbued cognitive dissonance that I have not yet embraced.

I will not miss all of them. Simply put, too many are intellectually lazy. Many of my colleagues think of the day they receive tenure as the last official day they have to produce research. They consider research as a burden, not as a labor of love that complements teaching.

As for the students, I know that I’ll miss the good ones. Any good professor treasures the joy of seeing in a student’s eyes the “ah-ha – now I get it” moment. It cannot be replicated, nor can it be easily described. It is sadly ever increasingly rare. In fact I think I am doing a genuine service to the better students by leaving. I cannot in good conscience dumb down a lecture, knowing full well that the gifted and talented have read four chapters beyond the syllabus, and that they are not being sufficiently challenged.

I am ready to move on – perhaps for a career where deadlines are honored, ideas are exchanged and gimmicks and fads are routinely avoided because they distract from advancing the mission of gaining and sharing knowledge. Yes, it is time to find another line of work, where I can enjoy the fruits of my labor, even if I realize that the grass is grayer, if not greener, elsewhere.

John Smith is the pseudonym of a professor at a liberal arts college. He asked to remain anonymous because he is continuing to teach while he is job-hunting and doesn’t want his comments to reflect on his institution.

[via Que Universidade?]

MAUS AERES

 

La science des quotas

André Gunthert, 7 Novembro 2008

Avec Valérie Pecresse arrive la science mise en quotas. Installée dans ses murs en mars 2007, l’agence pour l’évaluation de la recherche et de l’enseignement supérieur (AERES) a commencé ses travaux. Son site web montre de beaux locaux refaits à neuf dans un hôtel parisien chic. D’une main, le gouvernement fait miroiter des promesses de primes, de l’autre, il fait pleuvoir formulaires et demandes de rapport sur les chercheurs et les institutions. L’un des principaux instruments de cette politique du chiffre est un classement des revues scientifiques, à partir duquel on pourra étalonner chercheurs et laboratoires. Las, la publication prématurée d’une première liste au mois d’août a suscité la fronde des savants. Alors que le rôle des évaluateurs suppose une rigueur au-dessus de tout soupçon, les incohérences et les absences de cette nomenclature ont réveillé toutes les inquiétudes et constitué une bien piètre entrée en matière. Les articles et les protestations se multiplient, et une pétition demande le «retrait complet et définitif de la “liste des revues” de l’AERES».

Bref, on est ici dans du Sarkozy pur jus. De l’affichage et des moulinets de bras par devant, des outils inconsistants et beaucoup d’arrières-pensées par derrière. Le président de la République, on le sait, a la foi du charbonnier pour les indicateurs chiffrés. Mais il est plus facile de mettre en courbes des officiers de police que des experts de la mesure. Premiers utilisateurs des instruments statistiques, les savants sont bien placés pour savoir que ces images, plutôt que de traduire fidèlement la réalité, servent à choisir comment on veut la montrer (voir illustration).

Non que les chercheurs refusent de voir leur travail soumis à évaluation. Celle-ci est en effet leur condition quotidienne, lorsqu’ils soumettent un article à une revue ou un projet de recherche à un organisme. Mais la vision quantifiée de l’AERES n’a rien à voir avec cette expertise. La liste des revues les classe en trois rangs: A, B et C. Mais au lieu que le rang A donne un compte réel des meilleures publications internationales, on lui a fixé arbitrairement un quota de 25%. On comprend bien que la logique à l’oeuvre est strictement comptable. Le gouvernement-des-caisses-vides étant dans l’incapacité notoire de revaloriser le salaire des chercheurs (dont toutes les études s’accordent à dire qu’il est scandaleusement bas), il va répartir des lots de consolation aux plus méritants pour pouvoir afficher une politique de la recherche agressive. L’AERES en ses beaux locaux n’est qu’un outil pour camoufler la misère de la science.

Evaluer un chercheur à partir de ses lieux de publication est évidemment d’une grande bêtise. C’est un peu comme si on appréciait les qualités d’un conducteur en prenant pour critère la marque de sa voiture. Perdrai-je d’un coup mon acuité intellectuelle en publiant dans Romantisme (inconnu) plutôt que dans Les Annales (A)? Suis-je un meilleur savant quand je parais dans La Revue de synthèse (B) plutôt que dans Photographies (inconnu)? En réalité, la seule manière d’évaluer les travaux d’un chercheur, c’est de les lire. C’est aussi ce qu’admet implicitement l’AERES: sa méthode revient à utiliser le travail des comités de lecture, qui eux, ont procédé à cet examen. Le classement des revues n’est rien d’autre qu’un principe du coucou – une méta-expertise qui se défausse de l’évaluation réelle, et avoue simultanément l’impossibilité de s’acquitter de cette tâche.

On jugera du sérieux des outils de l’AERES en constatant qu’ils ne permettent pas de me définir comme “enseignant-chercheur publiant” (sic), au sens strict spécifié par l’agence (2 publications scientifiques de rang A au cours des quatre dernières années). Cela, non parce que je n’ai publié aucun article intéressant dans la période, mais parce que la plupart de mes lieux de publication ne sont pas recensés par la liste officielle (les chercheurs ont appris après sa mise en ligne que celle-ci était encore en cours de révision – ce qui fait du coup s’interroger sur la signification d’une publication incomplète). Dois-je modifier ma pratique de la recherche? Ne serait-ce pas plutôt à l’AERES d’admettre son incapacité à prendre en compte mon travail? L’agence chargée de l’évaluation devrait y prendre garde: avant les évalués, l’évaluation juge les évaluateurs.

L’AERES adosse son activité à un argument apparemment solide: celui de la promotion de l’excellence. Associé à l’accumulation d’autant d’erreurs, la répétition ad nauseam de ce discours a de quoi laisser perplexe. Mais il témoigne surtout d’un contresens sur la nature de la recherche. On peut organiser une compétition pour déterminer quels sont les meilleurs athlètes et les ranger par ordre d’arrivée sur un podium. Un sportif qui court le 100 mètres en plus de 11 secondes est exclu d’un tel concours – et un amateur qui l’effectue en plus de 12 n’est pas digne de chausser les crampons. Mais dans les sciences, aucun résultat de recherche n’est a priori plus profitable qu’un autre. Une modeste observation effectuée par un étudiant de première année peut être tout aussi intéressante que celle produite par un chercheur confirmé. On peut toujours récompenser tel spécialiste d’une discipline médiatique, mais la véritable utilité de la science est de maintenir des milliers de micro-communautés de par le monde, dont certaines ne comptent que quelques dizaines de membres, attachées à faire vivre et prospérer la moindre bribe de la curiosité humaine. Tel est est le sens du mot “recherche” – activité dont le principe est de ne se prévoir aucune fin ni limitation.

S’il y a de faux savants et de fausses découvertes, il n’y a pas de mauvaise science ni de recherches subalternes. Entonné par des ignorants, le discours de l’excellence va au rebours des traits les plus récents et les plus innovants de la pratique scientifique. Ce que l’on voit se mettre en place aujourd’hui sur les blogs des chercheurs du monde entier est une “garage science” bouillonnante et rapide, d’une redoutable efficacité. Dans ses tentatives de reconstitution du palmarès du concours général, l’AERES est on ne peut plus éloigné de l’agenda réel de la science qui se fait. L’évaluation à la française a-t-elle un avenir? Le moins qu’on puisse dire est qu’elle n’a pour l’instant pas convaincu.