I am working on a vocabulary-building book for SAT and GRE students. Below is a picture of the provisional cover of the book:
In order to have a wide corpus of classical texts to find word usage examples, I downloaded a massive ebook collection from the Gutenberg project and merged all of the text files into one big file that reached 12.4 gigabytes in size. I then wrote a PHP script that used the grep utility to search through about 250 billion lines of text1 to find the word usages I needed.
Here is an example of the results for the word “taciturn”:
In order to find interesting examples, I use the following regular expression:
egrep -aRh -B 20 -A 20 "\b(she|her|hers|his|he)\b.*taciturn" merged.txt
This finds usages of the word that start with a pronoun such as “she”. This helps find usages that occur mostly in novels, rather than other types of books (the Gutenberg collection contains many non-novel files, such as encyclopedias and legal works).
My first step toward speeding up the grep was to move the file to an old SSD I have that is attached to my desktop. The SSD supports up to 200 MB/second read speeds. This was not good enough, so I eventually moved it to my main Samsung SSD which has over 500 MB/second read speeds. Below is a screenshot of the iotop utility reporting a read speed of 447 M/s while grep is running:
My first idea to speed up the grep was to use GNU parallel or xargs, both of which allow grep to make use of multiple CPU cores. This was misguided since the limiting factor in this grep was not CPU usage but disk usage. Since my SSD is being maxed out, there is no point in adding more CPU cores to the task.
Using the following grep command, it took a little over 30 seconds to finish grepping the entire file once:
Here is the output for the time command which tells how long a command takes to finish:
One of the first suggestions I found is to prefix the command with LC_ALL=C, this tells grep to avoid searching through non-ANSI-C characters.
That seemed to make the grep very slightly faster:
Just to see what happens, I next used the fmt utility to reformat the file. The file currently is made up of short lines all separated by new lines. Using fmt, I changed it to having lines of 500 characters each. This was likely going to make the grep slower since it was going to match a lot more lines since the lines were going to be longer:
But on the upside, I was going to get a lot more results. The fmt command decreased the number of lines from 246 million to only 37 million:
But actually what happened when I did the next grep was that the grep time decreased to only 23 seconds:
I guess the reason is that grep has a lot fewer lines to go through.
Unfortunately it looked like fmt had corrupted the text. Here is an example:
I think the reason was that some (or most, or all) of the text files were using Windows-style newlines rather than Unix-style ones which was perhaps confusing fmt. So I used this command to convert all Windows-style newlines into spaces:
After that operation and running fmt again on the result, grepping again seems to result in non-corrupt results:
I also looked for the corrupted passage above to see how it looked now:
So it all seems fine now.
As far as I know there is no way to speed up the grep significantly further unless I get a lot of RAM and do the grep on a ramdisk, or get a much faster SSD. Just out of curiosity I decided to try out changing the fmt command to make lines of 1500 characters each to see how that affects the grep:
That didn’t actually do anything to speed up the grep further:
The study of hadith has been by and large restricted to hadith scholars. In this essay I present a fundamentally new approach to the study of hadith that combines the disciplines of traditional hadith criticism with uṣul al-fiqh (legal theory). For over a millennium, scholars of legal theory have discussed what makes certain hadiths mutawātir, or true beyond doubt, but these discussions have unfortunately been almost entirely ignored by hadith scholars. By combining legal theory with hadith criticism, we gain a powerful new empirical tool for judging the authenticity of hadiths that enables fiqh scholars to tap the vast resources of the science of hadith while implementing the long-neglected field of legal theory in the way hadiths are handled.
Counting the number of chains that a hadith possesses has been a traditional method of determining the strength of one hadith compared to another. Another method has been to determine the length of a chain. Hadiths with shorter chains were highly prized due to their higher authenticity. This method integrates both of these concepts: Hadiths with more chains and shorter chains receive higher percentage rankings than hadiths with fewer chains or shorter chains.
Update: The challenge
I have received a number of criticisms of this method, but unfortunately the vast majority of criticisms seem to arise from misunderstandings. For this reason I have devised a challenge for critics of this method: Present a single instance of two hadiths where the weaker hadith receives a higher percentage according to this method compared to the stronger hadith.
There are thousands of hadiths available to choose for this challenge. If the method is faulty, it should be extremely easy to find a case where the method leads to faulty results.
From binary to analog
It is rarely mentioned in scholarly circles and among the rest of the Muslim community that the categories that hadiths are placed into by hadith scholars, such as ṣaḥīḥ, are artificial, made-up categories. Some ṣaḥīḥ hadiths are vastly superior in their authenticity compared to other ṣaḥīḥ hadiths, and scholars are well aware of this.
It was natural and convenient for hadith scholars of the early and classical periods to view hadiths as being either ṣaḥīḥ or not-ṣaḥīḥ (ḥasan, ḍaʿīf, etc.). This is what I call binary thinking. It divides the world of hadith into two parts, one of which, ṣaḥīḥ, is assumed to be practically perfectly reliable despite the awareness that no two ṣaḥīḥ hadiths are ever equally reliable. It’s similar to dividing one’s friends and acquaintances into two categories, “trustworthy” and “untrustworthy”, despite the fact that we know the real world is much more complicated than that and that people, in reality, are on a spectrum of trustworthiness from most trustworthy to least trustworthy.
The essential innovation that my verification method introduces is that it supersedes the artificial, binary thinking of traditional hadith science by introducing an analog method of categorizing hadiths, suggested to me not by hadith scholars but by scholars of legal theory in their discussions of the reliability of hadiths, that far better represents reality. In the traditional method a hadith can either be ṣaḥīḥ or not-ṣaḥīḥ, while in my method a hadith can be 20% likely to be authentic, or 30%, or 90%. The method has room for representing all the various shades of authenticity that are hidden to ordinary Muslims when they read that a particular hadith is authentic.
This method helps free the great treasures of the science of hadith criticism that have so far been locked away in hadith collections and the works of hadith transmitter criticism (ʿilm al-rijāl) so that every Muslim can benefit from them by having that knowledge translated into an easy-to-understand mathematical ranking system.
Hadith and probability theory
Islam’s hadith literature (reports about the actions and sayings of the Prophet PBUH) is one of the most problematic aspects of the religion due to the issues concerning the reliability of transmitters. How do we know if a report going back six or seven generations to the Prophet PBUH truly and accurately reports what the Prophet PBUH said or did?
So far the science of hadith has largely limited itself to verifying the authenticity of hadiths by verifying the trustworthiness of each person in a hadith’s chain of narrators. If all the transmitters are trustworthy, the assumption is that the hadith is ṣaḥīḥ (“authentic” or “sound”) unless the hadith reports things that are clearly false or contradictory to other hadith narrations. The problem is that a hadith scholar’s own beliefs and biases can strongly affect whether they consider a strange and rare hadith to be authentic or not. A good example is the following hadith in Imām al-Bukhārīʾs collection:
Narrated Abu 'Amir or Abu Malik Al-Ash'ari that he heard the Prophet (ﷺ) saying, "From among my followers there will be some people who will consider illegal sexual intercourse, khazz (a type of clothing), the wearing of silk, the drinking of alcoholic drinks and the use of musical instruments, as lawful. And there will be some people who will stay near the side of a mountain and in the evening their shepherd will come to them with their sheep and ask them for something, but they will say to him, 'Return to us tomorrow.' Allah will destroy them during the night and will let the mountain fall on them, and He will transform the rest of them into monkeys and pigs and they will remain so till the Day of Resurrection."
(Ṣaḥīḥ al-Bukhārī 5590)
A non-scholar who reads this hadith will be greatly troubled by the implication that the use of musical instruments is a characteristic of misguided and impious Muslims. And since the hadith is authentic and present in Ṣaḥīḥ al-Bukhārī, they will face the difficult choice of either believing musical instruments (and hence all music) to be forbidden in Islam, or ignoring the hadith and going with the commonsense and widespread Muslim belief that music is permissible in Islam.
By introducing probability theory into the science of hadith, we gain an extremely powerful tool that enables us to judge just how seriously we should take any particular hadith. This is especially useful in the case of hadiths that seem to be contradicted by other hadiths. For example on the issue of musical instruments we have the following hadith:
Abu Bakr came to my house while two small Ansari girls were singing beside me the stories of the Ansar concerning the Day of Buath. And they were not singers. Abu Bakr said protestingly, "Musical instruments of Satan in the house of Allah's Messenger (ﷺ) !" It happened on the `Id day and Allah's Messenger (ﷺ) said, "O Abu Bakr! There is an `Id for every nation and this is our `Id."
Sahih al-Bukhari 952
An authentic version of this hadith (Sahih Muslim 892 b) tells us that the girls were using the instrument daff (“tambourine”). So here we have the Prophet PBUH approving of the use of musical instruments in his own home, yet the other hadith implies that musical instruments are wicked and unlawful.
Probabilistic hadith criticism helps solve the dilemma of having to choose between two hadiths that are both judged authentic by hadith scholars by telling us which one is stronger.
As it happens, the hadith mentioning the Prophet’s approval of musical instruments is far more “authentic” and believable than the hadith in which he disapproves of them.
In this essay I will use these hadiths on music as an illustration of the probabilistic hadith verification method.
Gathering the hadiths approving of music
The first step in hadith verification is to gather all existing versions of a hadith and their chains and to draw a diagram representing all of its transmitters. Below is a diagram of the version of the “two singing girls” hadiths found in Sahih al-Bukhari:
The blue box is the hadith, and the gray boxes are its transmitters. The first transmitter is Aisha, may God be pleased with her, wife of the Prophet PBUH. The second transmitter is Urwa b. al-Zubayr, her nephew. The third transmitter is Urwa’s son Hisham. The the fourth transmitter is the highly respected hadith scholar Hammad b. Usama b. Zayd. The fifth transmitter is Ubayd b. Ismail, a respected hadith transmitter. This transmitter gave the hadith to Imam al-Bukhari who recorded it.
The second version of the hadith is also in Sahih al-Bukhari:
That once Abu Bakr came to her on the day of `Id-ul-Fitr or `Id ul Adha while the Prophet (ﷺ) was with her and there were two girl singers with her, singing songs of the Ansar about the day of Buath. Abu Bakr said twice. "Musical instrument of Satan!" But the Prophet (ﷺ) said, "Leave them Abu Bakr, for every nation has an `Id (i.e. festival) and this day is our `Id."Sahih al-Bukhari 3931
Below is a diagram of this hadith’s chain:
You may note that the first three transmitters are the same as those of the previous hadith. So we can join their diagrams into one as follows:
Next is the chain found in Sahih Muslim 892 a:
This chain is exactly the same as the al-Bukhari’s first chain except for the last transmitter. We can therefore join it with the rest as follows:
We now therefore have three chains going back to the same hadith. Imam Muslim mentions two additional chains which we add to the diagram as follows:
We now move on to other collections that mention the same hadith. The first one is a version mentioned in Abu Uwaana’s Mustakhraj, which we add to the diagram as follows:
I have indicated the new chain in green. After further research, I was able to discover a further chain in Sahih al-Bukhari as indicated below:
Al-Bukhari mentions an alternative version of this new chain as indicated below:
The scholar Abu Uwana adds two of his own supporting chains to this version as follows:
We now have a fairly complete picture of all of the different chains of this hadith.
Adding the probabilities
To understand this section, some previous knowledge of probability theory will likely be required.
In order to start, we have to agree on an important assumption. How trustworthy is a single transmitter? This is a matter of intuition. After trying out various probabilities, I have settled on the value of 60%. This means that each trustworthy transmitter in a chain has a 60% chance of truthfully and correctly transmitting the hadith they transmit. This may seem low, but it actually works very well to represent the issues inherent in hadith transmitter verification.
This 60% is not a judgment on the transmitter’s character. It is a judgment on this specific piece of information in this specific place in the chain. It does not mean that a transmitter is 60% likely to be trustworthy. Rather, it is based on the uṣūl al-fiqh (Islamic legal theory) discussions on the number of witnesses needed for a certain piece of information to be considered mutawātir or true beyond doubt (see Wael B. Hallaq in Islamic Law and Jurisprudence: Studies in Honor of Farhat J. Ziadeh). It is general knowledge among scholars of legal theory that a single witness, no matter how trustworthy, can never impart certain knowledge. This is why Islamic law requires multiple witnesses in various legal cases. This requirement for multiple witnesses is not a judgment on the character of individual witnesses. It is a guarantee against various forms of error and uncertainty. According to the 60% assumption, four trustworthy witnesses mentioning the same piece of information side-by-side have a 97.44% chance of imparting it truthfully and correctly, while eight witnesses have a 99.93% chance, making a piece of information practically mutawātir, or widely-transmitted.
The 60% assumption takes into account the concept of signal decay. While a trustworthy witness who heard a piece of information today may have a 100% chance of imparting it correctly today, hadiths are transmitted between witnesses who may have held onto the information for decades. A hadith with four transmitters in its chain may in fact represent information passed down over four generations. There is an inherent signal decay in this multi-generational process, and the 60% assumption helps simulate this decay by requiring multiple transmitters at each point of the chain to maintain signal integrity.
According to probability theory, if you have a witness with 60% probability of truthfulness saying something, and then a second witness also with 60% probability of truthfulness comes along and says the same thing, the probability of both of them saying the truth increases, since we have two witnesses saying the same thing. We calculate this increase by multiplying the falsehood probabilities of the two persons as follows:
Person 1: 60% chance of truth = 40% chance of falsehood
Person 2: 60% chance of truth = 40% chance of falsehood
The chance of falsehood for both = 40% * 40% = 16%
The chance of truth of both = 100% - the chance of falsehood = 100% - 16% = 84%
Below is a diagram of an imaginary chain that represents these facts:
We have two people each of whom say that the Prophet PBUH said something. Each of them has a 60% probability of truthfulness and accuracy, but the two of them together have a 84% probability of truthfulness and accuracy, so the above imaginary hadith has an 84% chance of truthfulness and accuracy.
As we add more transmitters, the probability continually goes up:
Now we have four supporting witnesses, so their probability of truthfulness can be calculated as follows:
Falsehood probabilities: 40% * 40% * 40% * 40% = 2.56%
Truth probability: 100% - 2.56% = 97.44%
What this means is that if you have four Companions transmit the same hadith from the Prophet PBUH, the chance of the hadith being truly and correctly transmitted is 97.44%, which is a very high chance.
In probability theory 1 represents 100% and 0 represents 0. So another way of doing the above calculation is as follows:
0.4 * 0.4 * 0.4 * 0.4 = 0.0256
1 - 0.0256 = 0.9744
The benefits of the 60% assumption will become clear as we go through the verification process of the chains we gathered earlier. We will first deal with the top part of the diagram, which is as follows:
First, we will fill in the original 60% assumption by writing 0.6 in each bubble:
We do not add the 0.6 to the bubbles on the far right because those are written books, so they virtually have a 100% chance of reliability. What concerns us are unwritten, oral transmissions. The first step is to combine the three middle transmitters’ probabilities vertically, as follows:
Falsehood chances: (1-0.6) * (1-0.6) * (1-0.6) = 0.064
Truth chance: 1 - 0.064 = 0.936
Above, each of the three transmitters has a 100% – 60%, or 40% likelihood of falsely transmitting the hadith. But in order to get the probability of all of them falsely transmitting the hadith at the same time, we have to multiply these chances together: 40% * 40% * 40% = 6.4%. So the chance of all of them having falsely transmitted the hadith is only 6.4%, meaning that there is a 93.6% chance of them having truly and accurately transmitted this hadith.
We will call this “vertical combination”. We vertically combine all of the probabilities of the truthfulness and accuracy of the transmitters to get a single number, in this case 93.6%, which represents the authenticity of all them combined together.
We next have to do a horizontal combination. All three transmitters transmit the hadith from Hammad b. Usama, who also has a 0.6 (60%) chance of authenticity.
In the case of horizontal combination, rather than multiplying falsehood probabilities, we multiply truth probabilities. The reason is that as information passes down through a chain of transmitters each of whom have a 60% chance of authenticity, the chance of the information being correctly passed down decreases. There is more chance for error and fabrication. If you hear a Companion say that the Prophet PBUH said something, that is far more trustworthy than another person saying they heard their father say that their grandfather said that a Companion said that the Prophet PBUH said that.
So we multiply 0.93 by 0.6 to get 0.558, which means 55.8%. We now update the bubble for Hammad b. Usama to represent this new probability:
Hammad b. Usama’s probability went down from 0.6 to 0.558 because we did not hear the hadith directly from him, but from three people who claim to have heard him. Since those three people have a combined probability of 93% truthfulness and accuracy (rather than 100%), this slightly decreases the probability of the truthfulness and accuracy of the information we get from Hammad b. Usama.
We now move on to al-Bukhari’s second chain, which is as follows:
Using horizontal combination:
0.6 * 0.6 * 0.6 = 0.216
So this second chain has only a 21.6% probability of truthfulness and accuracy. The reason is that we do not have any supporting transmitters. We only have Muhammad b. al-Muthanna’s word for it that Muhammad b. Jaafar said that, and we only have Muhammad b. Jaafar’s word for it that Shu`ba said that. The deeper a chain goes back in time, the lower its probability of authenticity and accuracy becomes.
We next deal with two chains from Sahih Muslim:
Using vertical combination for the middle transmitters, we get a probability of 0.84 or 84%. Multiplying that by 0.6, we get 0.504, meaning that Abu Mu`awiyah has a 50.4% probability of authenticity.
Next we have the more interesting task of combining all the chains we examined above. We have to vertically combine the probabilities of Hammad (0.558), Shu`ba (0.216) and Abu Mu`awiyah (0.504):
Falsehood probabilities: (1-0.558) * (1-0.216) * (1-0.504) = 0.171877888
Truth probability: 1 - 0.171877888 = 0.828122112
So the probability of the truth of the information coming from these three transmitters is 82.8%.
Now we horizontally combine this with Hisham b. Urwa’s 0.6 probability, resulting in 0.496873267
We repeat these same steps for the remaining chains, as follows:
And by vertically combining the left-most probabilities (Hisham b. Urwa’s 49.68% and Muhammad b. Abd al-Rahman’s 21.04%), we get 0.602766552, so the information has a 60.27% probability of authenticity at this stage.
We next multiply this 0.602766552 by Urwa b. al-Zubayr’s 0.6 (horizontal combination), resulting in 0.361659931.
In the final step, we multiply this result by Aisha’s 0.6:
The result is 0.216995959. This means that the hadith with all of its chains has a 21.69% chance of authenticity and accuracy. This seems very low, but it is natural for a hadith that comes from a single Companion, through a single transmitter.
In my methodology, a hadith that reaches 20% or higher is ḥasan, a hadith that reaches 30% or higher is ṣaḥīḥ, a hadith that reaches 60% or higher is ṣaḥīḥ al-ṣaḥīḥ (a degree above ṣaḥīḥ), and a hadith that reaches 85% or higher is mutawātir (“widely transmitted”, a degree above ṣaḥīḥ al-ṣaḥīḥ).
Note that the method can easily be tweaked so that ṣaḥīḥ would start at 50% instead of 30%.. Instead of giving a transmitter in a chain a probability of 60%, we can give them a probability of 80%, which would increase the results arrived it. Note that it makes no difference what assumption we choose, as I have chosen 60%, for an individual transmitter, because the end result is always relative to other hadiths. Regardless of what assumption we choose, it is the case that a hadith with a stronger chain will receive a higher rank than a hadith with a weaker chain.
Different scholars may prefer different probabilities for what they consider ḥasan or higher. Personally these are the numbers I have settled on that my heart is comfortable with and accepts.
The hadith disapproving of music
We now move on to Ṣaḥīḥ al-Bukhārī 5590, the hadith that says misguided Muslims will approve of musical instruments. Below is a diagram of the hadith’s chain:
The hadith comes from a single chain that does not branch out, meaning it is an extremely questionable chain, which I call a “precarious” chain. To calculate its probability of authenticity, we horizontally combine all the 0.6 probabilities as follows:
Probability of truth = 0.6 * 0.6 * 0.6 *0.6 *0.6 *0.6 = 0.046656
This hadith therefore has a 4.66% chance of authenticity, which is extremely low. In my methodology a hadith between 10% and 20% is munkar (strange and likely to be false), while a hadith below 10% is ḍaʿīf (“weak”, i.e. almost certainly false). As discussed in another article, there is another reason to consider this hadith questionable: it says misguided Muslims will consider khazz (a type of clothing) lawful. But we know 20 Companions of the Prophet PBUH wore this type of clothing, so this makes the hadith even more dubious.
The hadith in which the Prophet PBUH says misguided Muslims will consider musical instruments lawful has only a 4.66% chance of authenticity, while the hadith in which the Prophet PBUH approves of musical instruments has a 21.69% chance of authenticity. Thus the hadith in which the Prophet PBUH approves of musical instruments is 4.6 times stronger than the hadith in which he disapproves of them (21.69 ÷ 4.66 = 4.65).
Based on this, we can confidently say that the evidence in support of the lawfulness of musical instruments is far stronger. Therefore approval of musical instruments is a vastly better representation of the Prophet’s sunna (tradition) than disapproval of them.
Defending the assumptions
The assumption that each transmitter has a 60% chance of accurately and truthfully transmitting the information needs to be defended. This assumption makes most ṣaḥīḥ narrations get an authenticity around 30%. If we treat these numbers empirically, it would mean that most authentic narrations are more likely to be false than true. For this reason I have chosen to treat all hadiths that reach 30% or higher as ṣaḥīḥ / authentic despite the fact that empirically they fall beneath the truth threshold of 50%.
My reasoning is that by choosing these assumptions, we create an interface between the empirical research world and the classical hadith studies world. A non-Muslim scholar may choose the 30% verdict on an authentic narration as a cause for skepticism, while a Muslim scholar can continue to make use of probability theory while adopting the traditional view of considering individual, low-truth-probability hadiths as authentic.
Another reason for the 60% assumption is that it gives us a very wide range of results. Some hadiths will reach close to 100%, many others will hover between 50% and 20%. This allows for representing a scholar’s empirical intuitions about these transmitted pieces of information, while also allowing them to remain in the classical hadith criticism world if they wish by choosing lower numbers, such as 30%, to represent full authenticity.
The study of Islam works on the basis of the autonomous consensus of the researchers, a concept I have defended elsewhere.1 Therefore introducing probability theory into hadith criticism will require the involvement of many researchers until a consensus emerges on the best assumptions to be chosen. Therefore my assumptions in this article are merely preliminary suggestions intended to illustrate what a probabilistic study of hadith would look like.
Combining probability theory and hadith criticism can greatly help in resolving issues surrounding contradictory hadith narrations by making it clear which hadiths are superior to which ones. It would be extremely helpful if we could build a new hadith collection that shows the probability of authenticity of each hadith rather than merely saying whether it is authentic or not. But such a work would require many years of work.
There is also the issue of sub-par transmitters. While I have chosen a probability 60% for reliable transmitters, for transmitters who are mudallis, non-qawī, non-ḥujja, or considered weak by some scholars and not others, lower probabilities will have to be used where necessary.
There are also different levels of weak transmitters. Some are considered weak for ideological reasons (because they held beliefs that hadith scholars considered unacceptable), while others are considered weak because they were caught lying or fabricating hadiths. Different probabilities will have to be used for different levels of weakness.
Combining the authenticity of multiple hadiths
One of the wonderful powers of this method of hadith verification is that it allows us to combine multiple hadiths related to the same topic in order to determine their combined authenticity. See this article on the hadiths related to the restriction on women traveling without a mahram where I illustrate the combination method.
Assume we have three hadiths, Hadith A (15%), Hadith B (20%), and Hadith C (25%), all of which mention the same ruling: women shouldn’t travel for more than three days without a mahram. The three hadiths come from different Companions and have completely different chains. We cannot use vertical combination because that’s for different chains of the same hadith. For various reasons, fabricating brand new hadiths that support an opinion we prefer is much easier than fabricating supporting chains the same hadith, therefore we have to use a combination process where the result we get is only half as strong as the result of combining different chains for the same hadith. The way we achieve this as follows:
Step 1: Vertically combine the authenticity hadiths as if they were different chains of the same hadith:
1-((1-0.15)×(1-0.2)×(1-0.25)) = 0.49
So if these hadiths had actually been different chains of the same hadith, their combined authenticity would have been 49%.
Step 2: Find the mathematical average of their original authenticity scores:
(0.15+0.2+0.25)÷3 = 0.2
So the average authenticity of the hadiths is 20%.
Step 3: Average the results of Step 1 and Step 2:
(0.49+0.2)÷2 = 0.345
The result is that the combined authenticity of the hadiths is 34.5%.
Like I’ve said before, the assumptions and calculation methods have to be defended. Personally my heart is content with this method of calculation for combining different hadiths. It fits with my instincts that different hadiths with authenticity scores of 15%, 20% and 25% should result in a 34.5% combined authenticity.
Further applications of the probabilistic method
See the articles on the Probabilistic Hadith Verification page for many studies by me in which I use the methodology developed in this article to verify hadiths on various important issues.
A famous hadith of the Prophet PBUH states:
I looked into Paradise and I saw that the most of its people were the poor; and I looked into the Fire and I saw that most of its people were women.Sahih al-Bukhari 3241
Note that this hadith does not actually say that the majority of women in Hell are women. It may just mean that the part of Hell that the Prophet PBUH was shown contained many women. None of the versions of the hadith say “the majority of people in Hell are women”. They all mention that when the Prophet saw Hell, he saw that the majority of the people (of the part he saw) were women. It is clear that the Prophet PBUH interpreted this vision as meaning that the majority are women. But since nothing in the Quran or hadith tells us that explicitly, we may consider it to be the Prophet’s own personal conclusion from what he saw.
Various versions of this statement are to be found in all of major hadith collections. I decided to conduct a study of all existing authentic versions of this hadith to find out just how reliable they are.
Below is the version from the Companion ʿImrān b. Ḥuṣayn [ra]:
All of the versions come through the single transmitter Abū Rajāʾ.
Next are the versions coming from the Companion Ibn Abbas [ra]:
The strongest chain comes again through the aforementioned transmitter Abū Rajāʾ. There is however an additional chain (the top one) coming through ʿAṭāʾ b. Yasār, through Zayd b. Aslam.
Next are the other chains coming from four other Companions:
None of these latter four chains are very strong because each comes through a single transmitter, through another single transmitter, before other witnesses come along.
In order to prove a point with reasonable certainty, a hadith should come from a binary tree chain, as follows:
The current hadith falls short of this standard as follows, with the red boxes indicating missing transmitters:
Besides the missing transmitters, the hadith also has the issue of having a duplicate witness (the yellow boxes). So in reality we only have two transmitters’ words for it that the two Companions said that.
However, the four additional chains are supporting evidence in favor of the hadith and cannot be ignored. Using a hadith verification methodology I have developed that uses probability theory to combine the reliability of each transmitter and chain to reach a single number that represents the chance of the hadith being true (see my essay about it), we perform the following calculation to combine all of the probabilities:
(1-(1−0.34)×(1−0.39015)×(1−0.0207)×(1−0.1076)×(1−0.0776)×(1−0.1716)) = 0.7312
So the verdict is that this hadith has a 73.12% probability of being truly from the Prophet PBUH, which is a very high probability for a hadith. In my highly stringent verification methodology, a hadith that reaches 60% or higher is ṣaḥīḥ al-ṣaḥīḥ (a degree above ṣaḥīḥ)
In conclusion, the Prophet PBUH almost certainly said that the majority of the people he saw in Hell were women. Whether this really means the majority are women, or whether only the part that he saw had a lot of women, we cannot say. So the hadith should not be used to imply that women are less pious or more evil than men.
There are numerous narrations that mention ruqya (the use of certain words, prayers or Quran recitations as charms or spells to heal or protect a person). I decided to to conduct a search of all major hadith collections and some minor ones to find all the hadiths that mention ruqya in order to find out just how authentic they are. I then used my own mathematical method of calculating hadith authenticity which combines probability theory with the science of hadith transmitter criticism (al-jarḥ wa-l-taʿdīl). The method (see my essay about it) is useful in judging between contradictory hadith narrations because it produces a single percentage for each hadith that reflects its chance of authenticity. We can then compare the chance of the authenticity of different hadiths to find out which one is most likely to be truly from the Prophet PBUH.
The hadith against ruqya has a 64.1% chance of authenticity, which makes it ṣaḥīḥ. The hadiths that support ruqya, however, are much lower in quality, the strongest having only a 20.78% chance of authenticity. But by combining the chance of the authenticity of all the ruqya-supporting hadiths, we reach a probability of 49.69%, which means that the crux of the meaning of the hadiths is likely to be true. It is strange, however, that the most authentic hadith in support of ruqya says that it is only to be used against the evil eye and scorpion stings.
In conclusion and considering all of the hadiths together, it appears that the Prophet PBUH forbade the use of ruqya in the pre-Islamic sense of casting a spell. But he permitted the use of the recitation of the Quran as a means of hopefully bringing about healing and protection. While some Muslims think that ruqya has an almost magical power that is guaranteed to bring about results, it is probably more correct to think of it as the same as prayer. It is merely the use of God’s words in the hope of attaining His blessings.
The traditional understanding of ruqya as casting spells is therefore highly doubtful and appears to be an importation of pre-Islamic Arab beliefs into Islam. The Prophet PBUH himself appears to have strongly disliked the spell-casting aspect of ruqya, which is why the most authentic narration speaks against it and mentions it along with other pre-Islamic practices. However, he appears to have tolerated the use of Quran recitation as a substitute for pre-Islamic forms of ruqya while allowing it to be called ruqya.
I am therefore fairly confident that we should reject the understanding of ruqya as spell-casting and instead think of it as similar to prayer and no more likely than prayer to be effective.
The hadith against ruqya
It is interesting to note that the most authentic narration on ruqya actually says good Muslims will not use it:
Verily the Messenger of Allah (ﷺ) said: Seventy thousand men of my Ummah would enter Paradise without rendering account. They (the companions of the Holy Prophet) said: Who would be those, Messenger of Allah? He (the Holy Prophet) said: They would be those who neither practise charm (ruqya), not take omens, nor do they cauterise, but they repose their trust in their Lord.Sahih Muslim 218 b
This hadith comes to us through three Companions (ignoring unauthentic chains). I decided to gather all of its versions from the major hadith collections to find out just how strong its chains are.
Below is a diagram of the chains coming through Ibn Abbas:
The numbers indicate probability of authenticity. Thus this chain has a 24% probability of being truly from Ibn Abbas (according to my methodology).
Below are the chains from Imran b. Husayn:
This chain is stronger and has a 45.7% probability of authenticity.
The last chain is from Ibn Masud and has a 12.85% chance of authenticity.
We use the following equation to combine all of these probabilities into one probability:
probability of authenticity = 1 - (1 - probability of authenticity of first chain) × (1 - probability of authenticity of second chain) × (1 - probability of authenticity of third chain chain) and so on.
1-((1−0.241)×(1−0.4579)×(1−0.12855)) = 0.641438499
The result is that this hadith has a 64.1% probability of authenticity. Any hadith that has a 60% probability of authenticity or higher is ṣaḥīḥ al-ṣaḥīḥ (a degree above ṣaḥīḥ) in my methodology, so this hadith is extremely authentic.
The hadiths in favor of ruqya
I asked `Aisha about treating poisonous stings (a snake-bite or a scorpion sting) with a Ruqya. She said, "The Prophet (ﷺ) allowed the treatment of poisonous sting with Ruqya."Sahih al-Bukhari 5741
It was narrated that Jabir said:Ibn Maja (authentic) Vol. 4, Book 31, Hadith 3515
“There was a family among the Ansar, called Al ‘Amr bin Hazm, who used to recite Ruqyah for the scorpion sting, but the Messenger of Allah (ﷺ) forbade Ruqyah. They came to him and said: ‘O Messenger of Allah! You have forbidden Ruqyah, but we recite Ruqyah against the scorpion’s sting.’ He said to them: ‘Recite it to me.’ So they recited it to him, and he said: ‘There is nothing wrong with this, this is confirmed.’”
The below diagram is the result of my search of all hadiths mentioning ruqya (click to enlarge it):
Below is a summary of the authenticity probabilities of the hadiths:
- Aisha 10.88%
- Aisha 7.77%
- Aisha 15.67%
- Aisha 7.278%
- Anas 14.29%
- Jabir’s uncle 1.2%
- Abu Saeed al-Khudri 11.27%
- Anas + Buryada + Imran 20.78%
- Jabir + Asmaa’ b. Umays 14.89%
- Awf b. Malik 5.8%
- Shifaa’ b. Abdullah 3.88%
The strongest hadith is the one coming from the Companions Anas, Burayda and Imran, and it is as follows:
The Prophet (ﷺ) said: No spell (ruqya) is to be used except for the evil eye or a scorpion sting.Sunan Abi Dawud 3884, Ibn Maja Vol. 4, Book 31, Hadith 3513, al-Tirmidhi Vol. 4, Book 2, Hadith 2057, etc.
The hadith, however, has only a 20.78% chance of authenticity, which is far below the 64.1% authenticity of the anti-ruqya hadith mentioned at the beginning.
There is one final step we can take by combining the authenticity probabilities of all the separate pro-ruqya hadiths, as follows:
1−(1−0.1088)×(1−0.077)×(1−0.1567)×(1−0.07278)×(1−0.1429)×(1−0.012)×(1−0.1127)×(1−0.2078)×(1−0.1489)×(1−0.058)×(1−0.0388) = 0.704960078
(0.1088+0.077+0.1567+0.07278+0.1429+0.012+0.1127+0.2078+0.1489+0.058+0.0388)÷11 = 0.1033
(0.704960078+0.1033)÷2 = 0.4041
Before merging the probabilities we divide each of them by two. This reflects the fact that we are combining entirely different hadiths together. It is easier to fabricate entirely new hadiths and chains than to fabricate supporting chains for the same hadith. So the probability of all the separate hadiths being true is lower than the probability of all the chains of the same hadith being true.
So the result is that there is a 40.41% chance that the crux of the meaning of the hadiths is true. In my methodology a hadith that reaches 30% or higher is ṣaḥīḥ. So the combined meaning of the hadiths together can be considered authentic.
AOA, Akhi! few days ago I met an aunt of me.She and her daughters are very social and they all are well known in their fields.My father couldn't afford our studies so we sisters are just graduate.Also my father never allowed us to go out much so we are kind of staying at home type girls.But Alhamdulillah all are married and happy in their lives.My aunt said to me that the kind of life u are living,is just making u a burden on society.So does a person must be recognized by society before dying?
Alaikumassalam wa rahmatullah,
In my opinion a saintly “soccer mom” who has no accomplishment beyond taking care of her family is infinitely more admirable than a selfish and greedy female CEO. A woman’s worth has nothing to do with her accomplishments and everything to do with her character. A saintly woman who carries out her duties (whatever they may be) is better than other women regardless of accomplishments.
The idea that a woman’s worth should depend on her accomplishments is a self-defeating modern superstition. It tells women they are not good enough unless they ignore their own desires and instincts and enter races with men in the corporate, political or scientific world.
I fully support women’s participation in these things. What I do not support is acting as if a woman’s worth depends on these things. It does not. Her worth depends on her character. I would consider a woman with no accomplishments but with a good character superior to a female Nobel Prize winner with a bad character any day.
Our accomplishments are gifts from God. He created us, gave us talents and made things easy for us. Acting as if accomplishments increase our worth is the height of arrogance, it is the same as a rich person thinking their money that God has given them makes them worthier than poor people.
I see nothing wrong with a woman having no interest in accomplishments and simply wanting to take care of her family. And I see nothing wrong with another woman who likes accomplishments. Neither is worthier than the other. Both are simply carrying out their duties.
It is only ignorance and arrogance that makes a scholar or scientist think their job is more glorious than a mother who takes care of her children. To me their worth depends on their character, including how well they try to carry out their duties. If God has enabled me or some woman to be a scholar and has put scholarship in our path, it would be shameful if we do not try to be the best scholars we can be. But if God has not enabled another person to become a scholar, then it is not shameful that they are not scholars.
I firmly believe that an uneducated and illiterate shepherd who fears God more than I do is a better and worthier person than I am regardless of my accomplishments.
Your aunt’s statement that you are a burden on society is rather ignorant and arrogant. Just because God made things easy for her and not for you makes her think she is better than you. If you fear God more than her and carry out your duties just as well as her, then you are superior to her even if she gains global fame in her field.
There is no worth, honor or glory except through God. Anyone who chases these things outside of God is chasing a mirage.
I do not want to discourage women from working in traditionally masculine fields. What I want to discourage them from is the arrogance to think that this makes them superior to other women. It does not. Whether you work with test tubes or diapers, you are a lowly servant of God and your only worth comes through Him. Anyone more pious and saintly than you is superior to you regardless of who you think you are.
And I find pious women who seek worth and honor through God to be infinitely more admirable than women who seek these things by trying to race with men in traditionally masculine fields. Of course there is nothing wrong with a woman wanting to work in these fields, what is wrong is her thinking this is something to be proud of. Like I said, being proud of your accomplishments is like being proud of being rich. Both are blessings from God that you would never have had if He had not made things easy for you. Accomplishments should only increase your humility and gratitude toward God.
If anyone, man or women, thinks their accomplishments makes them superior to someone more pious than them, then they have become arrogant and misguided. If you think your fame and accomplishments make you superior to a completely unknown mother who fears God more than you and whose only accomplishment is raising healthy and happy children, then that is the height of arrogance.
So never let someone make you think you are inferior just because they are more accomplished and famous than you. It is the same as letting a rich person make you think you are inferior because you are not as rich. Seek worth an honor only through God, He should be your standard and your guide, not other people. If you are more pious than your aunt, then she has absolutely nothing to be proud of, and her self-satisfaction has only set her up for failure in attaining God’s love and pleasure.
There is, however, the danger of letting our sense of our piety make us feel arrogant and superior to others. This too is wrong. Feeling superior to others is always wrong, whether because of piety, accomplishments or wealth. You should only compare yourself to what God wants you to be, and seeing your numerous failures in being the best person you can be in God’s sight should only increase your humility and fear of God’s dissatisfaction with you.