The title of this post really says it all. Here’s a quote from a document at the Editor Portal (sorry, I don’t have a public link for it), “AGU journals do not allow references to unpublished journal articles.” This includes JGR Space Physics.
Like the requirement of having open access to the data (observational or numerical) used to develop findings in a study, all scientific understanding on which the study is based (i.e., the cited literature) needs to be available. This does not mean freely available (the paper could be behind a subscription paywall) or even easily available (for instance, print only in an old monograph), but available somehow. Citing unpublished articles, especially the promissory note of “manuscript in preparation,” is forbidden.
Let me make an important clarification to this: unpublished articles cannot be cited in accepted or published AGU journal articles. At initial submission, citing papers that are “under review” or “accepted” is allowed. You need to provide a copy of the unpublished paper as a supplemental document so that the Editor and reviewers can see it and assess the worthiness of the reference. If they are not supplied, then reviewers can and should ask to see such references and the corresponding author should be ready to provide it. This means authors should confirm with the authors of the cited yet unpublished paper that it is okay to cite their paper and provide it to the Editor and reviewers.
On submission of any revisions, however, these other papers must have progressed to the level of being available online or in print. If not, then they should be removed and the manuscript revised to accommodate that change in referenced literature. If they are still in the revised manuscript, then AGU staff will ask the authors for a justification about the citation and will consult with the Editor about how to handle it. It could be that the other paper is close but not quite through to acceptance. If this can be verified, then we will probably let that through. It could be a companion paper or another paper in the same special section. Again, this is probably okay. If we let it remain, however, and the citing paper is accepted before the cited paper is available, then AGU/Wiley will hold the citing paper until the publication of that other one. If you must cite that paper, then your paper will wait until the other is available. If two papers mutually cite each other, AGU will coordinate publication. They will even coordinate with other publishers, like they did with the MAVEN special section in GRL last November, which came out simultaneously with 4 related papers in Science, all released in phase with a press conference.
For AGU journals, being “in press” means being available. AGU posts nearly all papers at the journal website within 3-4 days of acceptance. Other journals may or may not do this, though, so “in press” is not a guarantee that you can cite the paper. Like I said, AGU will contact other publishers and coordinate, release. Who knows, this might even expedite publication and availability of that other paper.
Finally, citation of some non-DOI references is allowed, especially those that are permanently archived. One example of special relevance to JGR Space Physics is the arxiv.org preprint service. Citing a paper there is allowed, even if it doesn’t have a corresponding peer-reviewed version available yet. Posting to arxiv.org is allowed because these papers are “permanently available” at this site. In the end, it is up to the Editor, in consultation with the authors and reviewers, to decide if the citation to a paper at arxiv.org (or similar service) is acceptable.
This has been enforced at GRL for a while but is relatively new for JGR Space Physics. If you start to see emails from AGU staff asking about these references to unpublished work, now you know why.
Notes from the JGR-Space Physics Editor-in-Chief 
The word “published” often is taken to be shorthand for “published in a peer-reviewed journal.” Thus, questions remain about conference proceedings. I hope that actual conference papers in books will continue to be citable (even if unrefereed). But what about presentations from meetings that didn’t have proceedings? There are many good ideas that were first presented at, say, an AGU or AAS meeting, and I’d want to refer to the very first presentation as a way to establish the priority. (Many of those ideas were followed up by papers later, but it may be important to note the earlier date/year of that first presentation.) However, the only publicly available piece of that presentation may be the abstract. That ought not erase our ability to cite the original presentation, though.
Good question. Published does not mean refereed, in this context, although I agree that it often is. Also, published does equal “freely available online.” If it’s a paper in a book, even unrefereed proceedings, then yes, you can cite it. Conference abstract? It depends if the abstract itself contains the useful information that makes it worthy of a reference. If the abstract says, “we did this,” and they were the first to do this, then perhaps that is enough merit for reference. I think this is a bit of a gray area and you could, at least, argue the case.
Historically, AGU has allowed citations to AGU abstracts in cases where a follow-up paper has not yet been published, but in most cases they prefer the full published version if one is available.
Conference proceedings books can be a tricky issue. Most such books are published through regular publishers that offer copies for library purchase. I have no objection to citing a paper from such a book that is available from my library, even if I have to physically go to the library to obtain it. But there are other conference proceedings that were never made available to people other than conference attendees–several of the early International Conference on Substorms proceedings are in this category. I would prefer that authors do not cite such papers because I have no way to obtain them, and would consider the inability to obtain such a paper adequate reason not to cite it even if a reviewer so requests.
I have occasionally needed to cite textbooks. These references may not be available online, but they should be available from the library, and as long as that is the case I see no reason to object to such a citation.